We use synthetic video and IMU data generated from the AMASS datasets (n = 500 subjects) to train deep learning models that can predict 3-D motion from noisy videos and/or uncalibrated IMUs.
Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home. We trained deep learning models that estimate human movement from noisy video data (VideoNet), inertial data (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration.
Data: We use a public dataset (https://amass.is.tue.mpg.de) to train the models. We also share the sample data and code for the demo run. This includes pre-trained models and one-subject sample video data (2D keypoints detections from 4 cameras). To run the demo code, please follow:
1. Download the demo.zip files in Downloads module and unzip it
2. Unzip SShin_TBME_demo.zip and models.zip in the directory
3. Install python package following the readme.txt
4. Run the demo code (run_demo.py) as instructed in the reademe file.
Code: https://github.com/CMU-MBL/FS_Video_IMU_Fusion
Citation: Shin, Soyong, Zhixiong Li, and Eni Halilaj. "Markerless Motion Tracking with Noisy Video and IMU Data." IEEE Transactions on Biomedical Engineering (2023).