Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes

Hitting snow Patting chair Patting table Rubbing chair Figure 1 . What sound does this object make when you strike it with your hand? We capture a 3D scene representation that can be used to simulate the sound that would result from a given hand motion. We reconstruct the scene Gaussian Splatting [20], then manipulate objects in the scene with hands, obtaining a sparse set of action-sound pairs. We use these examples to train a rectified flow model to map 3D hand trajectories at given position in a scene to a corresponding sound. At test time, a user can query an arbitrary 3D hand action and the model will estimate the resulting sound. Here we show several captured hand and audio pairs for two scenes (with representative video frames).