Origami Sensei: A Mixed Reality AI-Assistant

Learning creative hands-on skills, like origami, can be difficult for beginners. Conventional instructional methods often fail to support the experiential aspect of learning with timely and personalized feedback. Despite recent advancement of AI and Extended Reality in many fields, there is a lack of research on supporting learning in creative hands-on tasks. We investigate an AI-augmented Mixed Reality approach for learning hands-on creative tasks by introducing Origami Sensei as an approach for learning origami. Origami Sensei identifies the current step and relative locations of the paper using origami detection models, and projects real-time, personalized instructions directly onto the paper. We conducted a user study (n=18) comparing it with traditional video tutorials. Our findings show that participants prefer Origami Sensei, and it increases task efficiency and learner engagement. We introduce design insights for developing AI-augmented MR systems and highlight the potential for extending this approach to other creative hands-on tasks.