EMNLP Findings2023
IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, Babak Damavandi
Abstract
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with text and video, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videoswhile preserving the transitivity across these modalities. We introduce several new IMUbased Wearable AI applications such as motionbased media search, or an LM-based multimodal reasoning with motion sensor data -all using text as the grounding platform. In addition, we show that IMU2CLIP significantly improves downstream performances when finetuned for each application, demonstrating its universal usage as a new pre-trained resource. Our code and models will be released publicly.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SensorLM: Learning the Language of Wearable Sensors cites A Simple Framework for Contrastive Learning of Visual Representations
- SensorLM: Learning the Language of Wearable Sensors cites SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition
- SensorLM: Learning the Language of Wearable Sensors cites SensorChat: Answering Qualitative and Quantitative Questions during Long-term Multimodal Sensor Interactions
Cited by top-tier papers12
- SensorLM: Learning the Language of Wearable SensorsYuwei Zhang, Kumar Ayush, Siyuan Qiao, A. Ali Heydari et al.NeurIPS 2025 · 75 citations
- SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity RecognitionZechen Li, Shohreh Deldari, Linyao Chen, Hao Xue et al.EMNLP 2025 · 9 citations
- SensorChat: Answering Qualitative and Quantitative Questions during Long-term Multimodal Sensor InteractionsXiaofan Yu, Lanxiang Hu, Benjamin Z. Reichman, Dylan Chu et al.UbiComp 2025 · 4 citations
- Seeing without Pixels: Perception from Camera TrajectoriesZihui Xue, Kristen Grauman, Dima Damen, Andrew Zisserman et al.CVPR 2026 · 4 citations
- TxP: Reciprocal Generation of Ground Pressure Dynamics and Activity Descriptions for Improving Human Activity RecognitionLala Shakti Swarup Ray, Lars Krupp, Vítor Fortes Rey, Bo Zhou et al.UbiComp 2025 · 3 citations
Builds on2
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with LanguageAndy Zeng, Maria Attarian, Brian Ichter, Krzysztof Marcin Choromanski et al.ICLR 2023 · 171 citations
Related papers
- MoBind: Motion Binding for Fine-Grained IMU-Video Pose AlignmentDuc Duy Nguyen, Tat-Jun Chin, Minh HoaiCVPR 2026 · 1 citation
- ADAPT: Vision-Language Navigation with Modality-Aligned Action PromptsBingqian Lin, Yi Zhu, Zicong Chen, Xiwen Liang et al.CVPR 2022 · 45 citations
- CRIS: CLIP-Driven Referring Image SegmentationZhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao et al.CVPR 2022 · 337 citations
- CLIP2UDA: Making Frozen CLIP Reward Unsupervised Domain Adaptation in 3D Semantic SegmentationYao Wu, Mingwei Xing, Yachao Zhang, Yuan Xie et al.ACM MM 2024 · 12 citations
- Vistar: Enhancing the Perception Capability of LLMs under Imprecise IMU-Text AlignmentYatong Chen, Chenzhi Hu, Bowen He, Ruijie Wang et al.KDD 2026
