Lune

ICLR2025Top-tier venue

Towards Unified Human Motion-Language Understanding via Sparse Interpretable Characterization

Guangtao Lyu, Chenghao Xu, Jiexi Yan, Muli Yang, Cheng Deng

Abstract

Recently, the comprehensive understanding of human motion has been a prominent area of research due to its critical importance in many fields. However, existing methods often prioritize specific downstream tasks and roughly align text and motion features within a CLIP-like framework. This results in a lack of rich semantic information which restricts a more profound comprehension of human motions, ultimately leading to unsatisfactory performance. Therefore, we propose a novel motion-language representation paradigm to enhance the interpretability of motion representations by constructing a universal motion-language space, where both motion and text features are concretely lexicalized, ensuring that each element of features carries specific semantic meaning. Specifically, we introduce a multi-phase strategy mainly comprising Lexical Bottlenecked Masked Language Modeling to enhance the language model's focus on high-entropy words crucial for motion semantics, Contrastive Masked Motion Modeling to strengthen motion feature extraction by capturing spatiotemporal dynamics directly from skeletal motion, Lexical Bottlenecked Masked Motion Modeling to enable the motion model to capture the underlying semantic features of motion for improved crossmodal understanding, and Lexical Contrastive Motion-Language Pretraining to align motion and text lexicon representations, thereby ensuring enhanced crossmodal coherence. Comprehensive analyses and extensive experiments across multiple public datasets demonstrate that our model achieves state-of-the-art performance across various tasks and scenarios.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning cites AMASS: Archive of Motion Capture As Surface Shapes
  • Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • SGAR: Structural Generative Augmentation for 3D Human Motion Retrieval cites AMASS: Archive of Motion Capture As Surface Shapes
  • SGAR: Structural Generative Augmentation for 3D Human Motion Retrieval cites Language Models are Few-Shot Learners
  • SGAR: Structural Generative Augmentation for 3D Human Motion Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Semantically Comprehensive Token Pruning in LVLMs via Maximizing Concept Coverage cites Editing Is a Bargaining Game: Balanced Knowledge Editing in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. SGAR: Structural Generative Augmentation for 3D Human Motion RetrievalJiahang Zhang, Lilang Lin, Shuai Yang, Jiaying LiuNeurIPS 2025 · 7 citations
  2. Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor LearningHanmo Chen, Guangtao Lyu, Chenghao Xu, Jiexi Yan et al.ICML 2026 · 3 citations
  3. Editing Is a Bargaining Game: Balanced Knowledge Editing in Large Language ModelsChenghao Xu, Jiexi Yan, Muli Yang, Fen Fang et al.AAAI 2026
  4. Channel-masked Asymmetric Distribution Matching for Cross-Domain Generalized Dataset DistillationQi Liu, Chenghao Xu, Jiexi Yan, Guangtao Lyu et al.AAAI 2026
  5. Semantically Comprehensive Token Pruning in LVLMs via Maximizing Concept CoverageXueting Li, Qi Liu, Chenghao Xu, Xu Yang et al.ACL 2026

Builds on25

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  5. AMASS: Archive of Motion Capture As Surface ShapesNaureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll et al.ICCV 2019 · 1,784 citations

Related papers

  1. MSCode: Advancing Human Motion-Language Understanding via Modality-Shared CodebookHaoyu Shi, Huaiwen ZhangACL Findings 2026
  2. Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation LearningJiahang Zhang, Lilang Lin, Jiaying LiuACM MM 2023 · 26 citations
  3. LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and CaptioningZhe Li, Weihao Yuan, Yisheng He, Lingteng Qiu et al.ICLR 2025
  4. MotionGPT: Human Motion as a Foreign LanguageBiao Jiang, Xin Chen, Wen Liu, Jingyi Yu et al.NeurIPS 2023 · 698 citations
  5. Generative Motion Stylization of Cross-structure Characters within Canonical Motion SpaceJiaxu Zhang, Xin Chen, Gang Yu, Zhigang TuACM MM 2024 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines