Shape n' Swarm: Hands-on, Shape-aware Generative Authoring with Swarm UI and LLMs

This paper introduces a novel authoring method for swarm user interfaces that combines hands-on shape manipulation and speech to convey intent for generative motion and interaction. We refer to this authoring method as shape-aware generative authoring, which is generalizable to actuated tangible user interfaces. The proof-of-concept Shape n’ Swarm tool allows users to create diverse animations and interactions with tabletop robots by hand-arranging the robots and providing spoken instructions. The system employs multiple script-generating LLM agents that work together to handle user inputs for three major generative tasks: (1) thematically interpreting the shapes created by users; (2) creating animations for the manipulated shape; and (3) flexibly building interaction by mapping I/O. In a user study (n = 11), participants could easily create diverse physical animations and interactions without coding. To lead this novel research space, we also share limitations, research challenges, and design recommendations.