Spatial Intelligence for Physical AI

Sparse-view Multimodal Large 3D Reconstruction

We present a novel framework for large-scale indoor 3D reconstruction using sparse-view multimodal data to overcome the severe limitations that insufficient visual overlaps impose on conventional 3D Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines. By leveraging ubiquitously available ambient AIoT signals such as Wi-Fi and Bluetooth to provide coarse positional priors, our approach enables the initial assembly of local sub-models via per-cluster reconstruction before registering them onto the target space's floorplan to anchor them to the architectural layout.

Team Members

Prof. Shueng-Han Gary CHAN — Supervisor
Mr. Sizhe SONG — PhD
Mr. Joshua CHANG — MPhil
Mr. Ngoc Linh TA — MPhil

Core Technology & Architecture

We employ a generative model to perform inpainting for unobserved blind regions within the floorplan. During inpainting, structural cues are extracted from the floorplan to ensure the inpainted areas respect the real building structure. Our approach thus transforms sparse, disparate visual data into a coherent, seamless 3D model, offering a scalable solution for digital twinning and smart building applications. The synergy between signal-based localization and structure-aware generation marks a key advantage over purely visual reconstruction, achieving robustness and fidelity under extreme sparsity

Overview of the proposed framework pipeline. Starting from crowdsourced multimodal inputs (images, WiFi signals, and IMU data), the system first generates local 3D models, localizes and aggregates them onto the floor plan, and finally performs structure-aware inpainting to produce a coherent large-scale 3D model.

Key Features / Highlights:

AIoT Signal-Assisted Localization: Leverages ubiquitous indoor signals (such as Wi-Fi and Bluetooth) to supply coarse positional priors for sparse-view images, eliminating the need for the high visual overlap demanded by traditional SfM/MVS pipelines.
Divide-and-Conquer Pipeline: Adopts a "local modeling, global alignment" strategy that first builds local sub-models from signal-clustered images, then registers them onto the floorplan to establish consistent scale, orientation, and global relative poses.
Structure-Aware Generative Inpainting: Employs a generative model to fill unobserved blind spots on the floorplan while strictly enforcing architectural constraints, ensuring that all synthesized content remains physically plausible and architecturally correct.
Scalable Digital Twinning: Delivers a cost-effective, practical solution for smart building management, significantly lowering the barriers and expenses for large-scale indoor 3D modeling.

A demo of HKUST office areas. Three local 3D models are separately constructed first. They are then grounded onto the floorplan to form the global 3D model. A large indoor space can be tackled piece-by-piece like this through divide-and-conquer given crowd-sourced data.

Quantifying Advantages over Traditional Approaches

Traditional 3D reconstruction pipelines often struggle with prohibitive hardware costs, exhaustive manual labor, and structural drift. The comparative analysis below demonstrates how our framework overcomes these hurdles to deliver scalable, accurate indoor digital twins from casual, crowd-sourced inputs.

Feature / Dimension	Traditional Approach	Our Framework	Key Impact / Advantage
Hardware Requirements	Professional LiDAR scanners or specialized cameras	Casual, everyday smartphone cameras	Eliminates expensive hardware barriers, making 3D reconstruction accessible to anyone.
Data Collection & Labor	Exhaustive, full-scale manual site surveys	Crowd-sourced image and signal collection	Drastically reduces labor time, operational costs, and logistical friction on-site.
Structural Fidelity	Standard 3D generation (often prone to drifting or physical errors)	Structure-aware generation tied to floorplans	Ensures generated indoor models strictly respect real architectural layouts and remain physically plausible.

Visualizing the Reconstructed Shopping Mall
Demonstrating performance in a commercial environment, our pipeline processes sparse photos and architectural floorplans to generate a seamless 3D model.

Demonstration of our reconstruction pipeline on a shopping mall scene: Starting from the architectural floorplan (top-left) and sparse individual on-site photos (top-right), the framework processes the inputs to yield the final seamless, structure-aware inpainted full scene (bottom).

This video demo showcases the synthesized shopping mall environment, highlighting the seamless integration of sparse inputs into a continuous 3D scene.

Recent Developments

[Paper] SiteBuilder: Sitemap-Grounded 3D Completion for Atrium Scene (Accepted by ACMMM26, to appear)
[Deployment] Collaborative trials are now going on for massive commercial warehouses and local housing agents.