AIoT Rehabilitation
An Effective Multimodal Multiview 3D Human Mesh Approach

AIoT Rehabilitation

An Effective Multimodal Multiview 3D Human Mesh Approach

Rehabilitation requires continuous, personalized, and correctly performed training over weeks or months. Aligning with the 15th Five-Year Plan for AI-enabled healthcare, rehabilitation capacity, and smart elderly care, Hong Kong government has been promoting ageing-in-place, community-care, and applying AI for workforce efficiency. Under this backdrop, we research and develop a novel AIoT system to overcome the challenges of rehabilitation for healthy aging: to extend physiotherapists' professional supervision beyond limited face-to-face sessions and improve patient adherence to exercise from remote or at home. This addresses the limitations of existing tools, such as video uploads, wearables, or 2D cameras, which often struggle with depth, body angles, occlusion, and posture deviation.

Team Members

Core Innovation & Technology

In this project, we develop an innovative, rehabilitation-grade AIoT system based on multimodal, multiview 3D Human Mesh Estimation (3D HME). The flexible platform configures cameras, LiDAR, and mmWave modules to reconstruct patient mesh with privacy by design, comparing movements against therapist-defined standards or historical performance. It generates 3D heatmaps on deviations, range of motion, and large-model-assisted summaries to support physiotherapists review while protecting privacy. Backed by patent-pending research and industry support, we conduct multi-site trial validation across hospitals, centers, and homes to drive AI-driven rehabilitation, community care, care, healthy aging and industrial commercialization.

Edge AIoT Sensor
Clinical Deployment
User Interface

Key technical highlights:

  • Multiview multimodal uncalibrated AIoT sensors
  • 3D human mesh estimation (3D HME)
  • Privacy-preserving edge AI
  • VLM-based real-time feedback and patient data analytics

An integrated care solution for hospitals, centers and homes
An integrated care solution for hospitals, centers and homes
Human Mesh Estimation System Architecture
The Human Mesh Estimation System Architecture

Impact & Metrics

Empowering seamless remote care, the platform transforms rehabilitation through high-precision 3D tracking, privacy-preserving edge AI, and automated clinical insights.

  • A 3D HME analytics module for rehabilitation movement reconstruction, mesh comparison and movement-quality assessment, targeting a mean 3D keypoint deviation of ≤ 50 mm on selected controlled benchmark / annotated trial samples.
  • Modular sensing units covering cameras, LiDAR and optional mmWave modules, with different reference configuration profiles for home, clinic/community, and advanced prototype settings.
  • Efficient edge AI inference modules based on the team’s FasterNet-related research for real-time operation of ≥ 25 FPS.
  • Compact edge-computing prototype terminals for local multimodal data processing and rehabilitation indicators reducing routine raw-video transmission.
  • Physiotherapist review tools presenting 3D heatmaps, movement deviations, completion quality, stability, progress indicators and LLM-assisted summaries.

Multiview Camera-Based 3D HME ​

Milestones & Status

 
Prev
Next