Lune

CVPR2022Top-tier venue

Learning Video Representations of Human Motion from Synthetic Data

Xi Guo, Wei Wu, Dongliang Wang, Jing Su, Haisheng Su, Weihao Gan, Jian Huang, Qin Yang

Abstract

In this paper, we take an early step towards video representation learning of human actions with the help of large-scale synthetic videos, particularly for human motion representation enhancement. Specifically, we first introduce an automatic action-related video synthesis pipeline based on a photorealistic video game. A large-scale human action dataset named GATA (GTA Animation Transformed Actions) is then built by the proposed pipeline, which includes 8.1 million action clips spanning over 28K action classes. Based on the presented dataset, we design a contrastive learning framework for human motion representation learning, which shows significant performance improvements on several typical video datasets for action recognition, e.g., Charades, HAA 500 and NTU-RGB. Besides, we further explore a domain adaptation method based on cross-domain positive pairs mining to alleviate the domain gap between synthetic and realistic data. Extensive properties analyses of learned representation are conducted to demonstrate the effectiveness of the proposed dataset for enhancing human motion representation learning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Progressive Transformation Learning for Leveraging Virtual Images in Training cites A Simple Framework for Contrastive Learning of Visual Representations
  • Do Generated Data Always Help Contrastive Learning? cites A Simple Framework for Contrastive Learning of Visual Representations
  • Is Space-Time Attention All You Need for Video Understanding? cites STM: SpatioTemporal and Motion Encoding for Action Recognition
  • Is Space-Time Attention All You Need for Video Understanding? cites SlowFast Networks for Video Recognition
  • STM: SpatioTemporal and Motion Encoding for Action Recognition cites SlowFast Networks for Video Recognition
  • ConsisDrive: Identity-Preserving Driving World Models for Video Generation by Instance Mask cites InstaDrive: Instance-Aware Driving World Models for Realistic and Consistent Video Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Do Generated Data Always Help Contrastive Learning?Yifei Wang, Jizhe Zhang, Yisen WangICLR 2024 · 36 citations
  2. InstaDrive: Instance-Aware Driving World Models for Realistic and Consistent Video GenerationZhuoran Yang, Xi Guo, Chenjing Ding, Chiyu Wang et al.ICCV 2025 · 3 citations
  3. ConsisDrive: Identity-Preserving Driving World Models for Video Generation by Instance MaskZhuoran Yang, Yanyong ZhangICLR 2026 · 2 citations
  4. Progressive Transformation Learning for Leveraging Virtual Images in TrainingYi-Ting Shen, Hyungtae Lee, Heesung Kwon, Shuvra S. BhattacharyyaCVPR 2023
  5. Less is More: Data-Efficient Adaptation for Controllable Text-to-Video GenerationShihan Cheng, Nilesh Kulkarni, David Hyde, Dmitriy SmirnovCVPR 2026

Builds on13

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. SlowFast Networks for Video RecognitionChristoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming HeICCV 2019 · 4,104 citations
  3. Is Space-Time Attention All You Need for Video Understanding?Gedas Bertasius, Heng Wang, Lorenzo TorresaniICML 2021 · 2,927 citations
  4. TSM: Temporal Shift Module for Efficient Video UnderstandingJi Lin, Chuang Gan, Song HanICCV 2019 · 2,049 citations
  5. STM: SpatioTemporal and Motion Encoding for Action RecognitionBoyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu et al.ICCV 2019 · 442 citations

Related papers

  1. Video Representation Learning with Graph Contrastive AugmentationJingran Zhang, Xing Xu, Fumin Shen, Yazhou Yao et al.ACM MM 2021 · 6 citations
  2. Action-Conditioned 3D Human Motion Synthesis with Transformer VAEMathis Petrovich, Michael J. Black, Gül VarolICCV 2021 · 672 citations
  3. Spatio-temporal Contrastive Domain Adaptation for Action RecognitionXiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yue et al.CVPR 2021
  4. Composable Augmentation Encoding for Video Representation LearningChen Sun, Arsha Nagrani, Yonglong Tian, Cordelia SchmidICCV 2021 · 20 citations
  5. Rethinking Masked Data Reconstruction Pretraining for Strong 3D Action Representation LearningTao Gong, Qi Chu, Bin Liu, Nenghai YuAAAI 2025 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines