DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References

State n Goal n+1 Object Neural Tracking Controller Action Environment Result n (State n+1) … … (b) Mimicking intricate manipulations with thin objects and intriguing in-hand re-orientations Timestamp (c) Tracking noisy interactions with unreachable goal states and real-world evaluations Timestamp Kinematic References Results Kinematic References Results Goal 0 Goal n Goal n+1 Kinematic References Observation at timestep n Timestamp (a) Inference flow of the neural tracking controller Update Goal n+2 Figure 1: DexTrack learns a generalizable neural tracking controller for dexterous manipulation from human references. It generates hand action commands from kinematic references, ensuring close tracking of input trajectories (Fig. (a)), generalizes to novel and challenging tasks involving thin objects, complex movements and intricate in-hand manipulations (Fig. (b)), and demonstrates robustness to large kinematics noise and utility in real-world scenarios (Fig. (c)). Kinematic references are illustrated in orange rectangles and background.