AI of Oz: Enhancing Wizard of Oz Studies in HCI with AI Assistance for Human Moderation

The Wizard of Oz (WoZ) method is a common and popular approach for simulating interactive systems in Human-Computer Interaction (HCI). Running such studies is demanding for researchers because the human wizard must manage human–agent interactions in real time while keeping participants safe and the interaction natural. Many WoZ systems struggle to reproduce complex agent behaviours without minimal delays or heavy workload for the moderator. We introduce AI of Oz, a framework that uses large language models to support researchers by monitoring ongoing interactions, detecting sensitive moments, and suggesting contextually appropriate responses. In a study with 20 HCI-related researchers, the system improved participants’ ability to manage interactions and maintain control compared to a version without AI support. We outline implications for WoZ research and note current limitations and future directions.