Interaction Through Instruments: Extending Surgical Instruments as Interaction Devices

Interaction while performing physical tasks is inherently challenging, as both hands are fully engaged. In Minimally-Invasive Surgery (MIS), for instance, navigating images requires either delegation, causing frustration and delays, or hand de-sterilization, increasing risk. We introduce Interaction Through Instruments, an interaction paradigm in which task instruments become interaction devices. To design this technique in MIS, we first conduct a survey (N=23) identifying intraoperative needs, interaction strategies and workarounds, and persistent challenges. Then, through five participatory design workshops (N=10), we identify challenges in blending a user interface into views of a physical space, informing the design of InteractOR, a system that combines surgical instrument segmentation with pinch-gesture recognition to enable interaction within the surgical view. Finally, in a Comparative Structured Observation study (N=12) we compare two visualization strategies (side-by-side and overlay) against delegation, showing that interaction through instruments can reduce focus shifts, increase efficiency, and foster autonomy.