HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos
Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard A. Newcombe, Robert Wang
Abstract
Meta Reality Labs facebookresearch.github.io/hot3d Figure 1. HOT3D overview. The dataset includes multi-view egocentric image streams from Aria [13] and Quest 3 [40] annotated with high-quality ground-truth 3D poses and models of hands and objects. Three multi-view frames from Aria are shown on the left, with contours of 3D models of hands and objects in the ground-truth poses in white and green, respectively. Aria also provides 3D point clouds from SLAM and eye gaze information (right).
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Cited by top-tier papers35
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric VideoRyan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu et al.ICLR 2026 · 248 citations
- Vision-Language-Action Pretraining from Large-Scale Human VideosHao Luo, Yicheng Feng, Wanpeng Zhang, Sipeng Zheng et al.ICML 2026 · 104 citations
- 4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular VideosZhen Xu, Zhengqin Li, Zhao Dong, Xiaowei Zhou et al.NeurIPS 2025 · 51 citations
- H-RDT: Human Manipulation Enhanced Bimanual Robotic ManipulationHongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan et al.AAAI 2026 · 25 citations
- UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human VideosGu Zhang, Qicheng Xu, Haozhe Zhang, Jianhan Ma et al.CVPR 2026 · 23 citations
Builds on15
- Depth Anything V2Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao et al.NeurIPS 2024 · 2,305 citations
- FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB ImagesChristian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan C. Russell et al.ICCV 2019 · 493 citations
- Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose EstimationShreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, Vincent LepetitCVPR 2022 · 155 citations
- OnePose: One-Shot Object Pose Estimation without CAD ModelsJiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He et al.CVPR 2022 · 153 citations
- BEHAVE: Dataset and Method for Tracking Human Object InteractionsBharat Lal Bhatnagar, Xianghui Xie, Ilya A. Petrov, Cristian Sminchisescu et al.CVPR 2022 · 144 citations
Related papers
- Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine PerceptionXiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters et al.ICCV 2023 · 145 citations
- SHOW3D: Capturing Scenes of 3D Hands and Objects in the WildPatrick Rim, Kevin Harris, Braden Copple, Shangchen Han et al.CVPR 2026 · 5 citations
- Can We Infer Object Pose Changes from Hand Movements?Julien Berry, Emmanuel Pietriga, Olivier Chapuis, Caroline AppertCHI 2026 · 1 citation
- EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUsZhen Fan, Peng Dai, Zhuo Su, Xu Gao et al.AAAI 2025 · 13 citations
- H2O: Two Hands Manipulating Objects for First Person Interaction RecognitionTaein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo et al.ICCV 2021 · 271 citations
