Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

Surgical navigation has proven to be an effective approach for providing real-time guidance and visualization of relevant information by estimating the pose of the patient’s anatomy and surgical tools. During navigated surgery, instruments are commonly equipped with fiducial markers and tracked by stationary optical tracking systems (OTS) to provide accurate navigation cues. Augmented Reality (AR) has been adopted for intuitive visual guidance, even motivating several efforts to enable surgical instrument tracking through built-in sensors on Head-Mounted Displays (HMDs). However, existing tracking methods typically require a direct line-of-sight to instruments, which is challenging to maintain in dynamic surgical environments due to frequent occlusions caused by moving medical equipment, surgical tools, and personnel. To address this challenge, this work introduces a novel framework capable of tracking surgical instruments even under occlusion by fusing different sensors in a dynamic scene graph representation. Our framework uniquely combines tracking systems with varying degrees of accuracy, providing real-time assessments of tracking reliability to the user. Unlike conventional sensor fusion approaches that are heavily dependent on specific sensor modalities, the proposed method is agnostic to tracking device modality and robust to their motion states (e.g., stationary OTS versus dynamic AR-HMD). Experimental results demonstrate that our dynamic scene graph framework successfully integrates and optimizes measurements from multiple tracking sources, significantly enhancing AR visualization consistency and accuracy with robustness under occlusion.