Reconstructing Hand-Object Interactions in the Wild
Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, Jitendra Malik
Abstract
We study the problem of understanding hand-object interactions from 2D images in the wild. This requires reconstructing both the hand and the object in 3D, which is challenging because of the mutual occlusion between the hand and the object. In this paper we make two main contributions: (1) a novel reconstruction technique, RHO (Reconstructing Hands and Objects), which reconstructs 3D models of both the hand and the object leveraging the 2D image cues and 3D contact priors; (2) a dataset MOW (Manipulating Objects in the Wild) of 500 examples of hand-object interaction images that have been "3Dfied" with the help of the RHO technique. Overall our dataset contains 121 distinct object categories, with a much greater diversity of manipulation actions, than in previous datasets.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 2c858e9d-f522-4f9e-aa9a-0b1cfb602989Cited by top-tier papers86
- Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose EstimationShreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, Vincent LepetitCVPR 2022 · 155 citations
- MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular ImageXingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang et al.CVPR 2022 · 97 citations
- ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and SynthesisLixin Yang, Kailin Li, Xinyu Zhan, Jun Lv et al.CVPR 2022 · 82 citations
- Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction ClipsYufei Ye, Poorvi Hebbar, Abhinav Gupta, Shubham TulsianiICCV 2023 · 80 citations
- OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object InteractionLixin Yang, Kailin Li, Xinyu Zhan, Fei Wu et al.CVPR 2022 · 79 citations
Builds on10
- FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB ImagesChristian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan C. Russell et al.ICCV 2019 · 493 citations
- Resolving 3D Human Pose Ambiguities With 3D Scene ConstraintsMohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, Michael J. BlackICCV 2019 · 384 citations
- Coherent Reconstruction of Multiple Humans From a Single ImageWen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou et al.CVPR 2020
- Mesh R-CNNGeorgia Gkioxari, Justin Johnson, Jitendra MalikICCV 2019
- Understanding Human Hands in Contact at Internet ScaleDandan Shan, Jiaqi Geng, Michelle Shu, David F. FouheyCVPR 2020
Related papers
- HOnnotate: A Method for 3D Annotation of Hand and Object PosesShreyas Hampali, Mahdi Rad, Markus Oberweger, Vincent LepetitCVPR 2020
- Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the WildGyeongsik MoonCVPR 2023
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the WildYumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu et al.CVPR 2025
- HandCraft: Tactile-Informed Hand-Object Dynamics Capture and Realistic RenderingHongyang Lin, Kuixiang Shao, Peijun Xu, Zhuoyang Bu et al.ACM MM 2025
- Reconstructing In-the-Wild Open-Vocabulary Human-Object InteractionsBoran Wen, Dingbang Huang, Zichen Zhang, Jiahong Zhou et al.CVPR 2025
