Perceive, Adapt, and Interact: An Intelligent Virtual Agent for Education in Mixed Reality

Mixed Reality (MR) environments are reshaping education by fostering engagement and motivation. Within such environments, Intelligent Virtual Agents (IVAs) can serve as personalized guides, tutors, or collaborative partners. In this research, we present a novel IVA for education that perceives the learner's environment, adapts its guidance to their progress, and interacts with both the learner and the real world. Our multimodal IVA integrates speech capabilities powered by large language models (LLMs) with computer vision for dynamic scene understanding, allowing it to converse naturally with learners and provide adaptive support based on their progress. To evaluate our IVA, we implemented a Scratch programming scenario using the Magic Leap 2 headset. We conducted a user study with 24 university students (12 male, and 12 female) who completed the same tasks using three systems: basic AR cues, a chatbot with AR cues, and our IVA. Quantitative data were obtained using standardized questionnaires such as the Technology Acceptance Model (TAM), the User Engagement Scale (UES), and the System Usability Scale (SUS), alongside qualitative feedback. The results revealed that although the IVA system took a longer time compared to other systems, it fostered greater motivation and engagement, achieved the highest SUS score, and showed the strongest TAM correlations. Participants perceived IVA as the most valuable system, highlighting the potential of multimodal embodied agents as virtual tutors in MR.