Investigating the Impact of Voice-only and Embodied Conversational Virtual Agents on Mixed Reality Puzzle Solving

Conversational Virtual Agents (CVAs) offer a promising approach for enhancing user task performance in Mixed Reality (MR) environments. This paper explores the integration of a CVA into an MR application designed to assist in solving a 2D physical puzzle, offering enhanced spatial cognitive capabilities. Using the CVA classification architecture and the MiRAS (Mixed Reality Agents) Cube Taxonomy, we developed an MR system with a state-aware assistant to guide users in a puzzle-solving task only when requested. The primary research question is whether or not the CVA needs to be embodied. We conducted a study with 34 participants to investigate the influence of Voice-only and Embodied CVAs on puzzle-solving performance, user interactions with the assistant, the assistant’s social presence, overall system cognitive workload, and users’ perceptions of future system use. Both modalities showed equivalent outcomes regarding the number of position- and orientation-related queries, perceived usability, message and affective understanding, performance, frustration, and usefulness. However, results showed that Voice-only CVA significantly enhanced task efficiency: participants completed puzzles more quickly and accurately, reporting lower effort than in the Embodied condition. These findings suggest that Voice-only CVAs may be more effective for tasks like puzzle solving, where auditory guidance alone appears sufficient to support better performance.