From Structure to Semantics: Hypergraph-Based AR Assembly Guidance with LLM-Mediated Narration

Effective Augmented Reality (AR) guidance for complex assembly faces a dual challenge: the inability of conventional liaison graphs to represent procedural logic, and the cognitive burden imposed by visual instructions. We argue that the solution requires a more expressive structure to overcome these representational deficits and a narration approach to mediate instruction complexity. Our method first employs an assembly hypergraph to capture the task's hierarchical information, from which an A* search algorithm generates an optimal assembly path. Then a Large Language Model (LLM)-mediated narration workflow is designed to address the ergonomic deficiencies of the machine-centric path. It employs an optimizer to improve fluency, followed by a narrator that crafts the steps into an intuitive instruction narration. A within-subjects user study (N = 24) revealed a progressive enhancement from our method's components. The transition from a liaison-graph baseline to the hypergraph alone improved objective outcomes by reducing task time and errors and improving subjective ratings (SUS, NASA-TLX, TAM, and ARI). Subsequently, augmenting the LLM-mediated narration maintained these gains while lowering cognitive load and elevating user experience and usability. Our findings indicate the value of our AR assembly design and discuss the opportunities of using LLM as a mediation layer for better user interaction.