Real-Time Multimodal Fingertip Contact Detection via Depth and Motion Fusion for Vision-Based Human–Computer Interaction

Precise fingertip contact detection is a fundamental challenge for natural and immersive virtual reality (VR) interaction, yet existing vision-based methods suffer from insufficient accuracy, with typical depth errors (12-25 mm) far exceeding the <3 mm precision required to reliably distinguish hovering from true contact. While commercial motion-capture systems provide sub-millimeter accuracy, their prohibitive cost limits accessibility. To address this gap, we present a highly accurate and cost-effective fingertip contact detection system based on a single RGB camera. We introduce a specialized dataset of 53,300 RGB-depth pairs capturing millimeter-scale hand-table typing interactions and systematically fine-tune seven stateof-the-art depth estimation architectures. This reduces the mean absolute error (MAE) by 68%, from 12.3 mm to 3.84 mm, achieving 95.9% depth accuracy (δ 1 ) and a 94.4% contact-detection F1 score. Our system enables typing speeds of 45.6 WPM with a 3.1% character error rate under controlled conditions (38.3 WPM, 5.3% CER for firstsession novice users), approaching the performance of dedicated depth-sensing hardware and commercial VR/AR input systems while requiring only a standard RGB camera. These results demonstrate that, for specialized close-range HCI tasks, domain-specific fine-tuning on targeted datasets can outperform architectural innovation, helping democratize high-precision hand tracking. The dataset, training pipeline, evaluation scripts, and pretrained models are publicly available for research use. 1 1 https : / / muxiddin19 . github . io / Multimodal - Fingertip -Contact -Detection -via -Depth -and - Motion-Fusion