2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos

When interacting with objects, humans effectively reason about which regions of objects are viable for an intended action, i.e., the affordance regions of the object. They can also account for subtle differences in object regions based on the task to be performed and whether one or two hands need to be used. However, current visionbased affordance prediction methods often reduce the problem to naive object part segmentation. In this work, we propose a framework for extracting affordance data from human activity video datasets. Our extracted 2HANDS dataset contains precise object affordance region segmentations and affordance class-labels as narrations of the activity performed. The data also accounts for bimanual actions, i.e., two hands co-ordinating and interacting with one or more objects. We present a VLM-based affordance prediction model, 2HandedAfforder, trained on the dataset and demonstrate superior performance over baselines in affordance region segmentation for various activities. Finally, we show that our predicted affordance regions are actionable, i.e., can be used by an agent performing a task, through demonstration in robotic manipulation scenarios. Project-website: sites.google.com/view/2handedafforder Dataset Image type & source # Images Affordance data Annotation source Annotation type # Aff. classes # Obj. classes Class-labels Bimanual IIT-AFF [34] Exocentric [44] 8.8K Manually-labeled Masks 9 10 Explicit No AGD20K [31] Exo+Egocentric [3, 27] 23.8K Manually-labeled Heatmaps 36 50 Explicit No 3DOI [38] Exo+Egocentric [5, 39] 10K Manually-labeled Points 3 n.a. Explicit No ACP [12] Egocentric [5] 15K Auto-labeled Heatmaps n.a. n.a. none No VRB [1] Egocentric [5] 54K Auto-labeled Heatmaps n.a. n.a. none No 2HANDS Egocentric [5] 278K Auto-labeled Precise Masks 73 163 Narrations Yes