Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality

Autonomous articial intelligence (AI) agents have emerged as promising protocols for automatically understanding the language-based environment, particularly with the exponential development of large language models (LLMs). However, a ne-grained, comprehensive understanding of multimodal environments remains under-explored. This work designs an autonomous workow tailored for integrating AI agents seamlessly into mixed reality (MR) applications for ne-grained training. We present a demonstration of a multimodal ne-grained training assistant for LEGO brick assembly in a pilot MR environment. Specically, we design a cerebral language agent that integrates LLMs with memory, planning, and interaction with MR tools and a vision-language agent, enabling agents to decide their actions based on past experiences. Furthermore, we introduce LEGO-MRTA, a multimodal ne-grained assembly dialogue dataset synthesized automatically in the workow served by a commercial LLM. This dataset comprises multimodal instruction manuals, conversations, MR responses, and vision question answering. Last, we present several prevailing open-resource LLMs as benchmarks, assessing their performance with and without ne-tuning on the proposed dataset. We anticipate that the broader impact of this workow will advance the development of smarter assistants for seamless user interaction in MR environments, fostering research in both AI and HCI communities.