EMNLP Findings2025
Automating eHMI Action Design with LLMs for Automated Vehicle Communication
Ding Xia, Xinyue Gui, Fan Gao, Dongyuan Li, Mark Colley, Takeo Igarashi
Abstract
The absence of explicit communication channels between automated vehicles (AVs) and other road users requires the use of external Human-Machine Interfaces (eHMIs) to convey messages effectively in uncertain scenarios. Currently, most eHMI studies employ predefined text messages and manually designed actions to convey these messages, which limits the real-world deployment of eHMIs, where adaptability in dynamic scenarios is essential. Given the generalizability and versatility of large language models (LLMs), they could potentially serve as automated action designers for the message-action design task. To validate this idea, we make three contributions: (1) We propose a pipeline that integrates LLMs and 3D renderers, using LLMs as action designers to generate executable actions for controlling eHMIs and rendering action clips. (2) We collect a user-rated Action-Design Scoring dataset comprising a total of 320 action sequences for eight intended messages and four representative eHMI modalities. The dataset validates that LLMs can translate intended messages into actions close to a human level, particularly for reasoning-enabled LLMs. (3) We introduce two automated raters, Action Reference Score (ARS) and Vision-Language Models (VLMs), to benchmark 18 LLMs, finding that the VLM aligns with human preferences yet varies across eHMI modalities. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on7
- Language Models Meet World Models: Embodied Experiences Enhance Language ModelsJiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu et al.NeurIPS 2023 · 180 citations
- Color and Animation Preferences for a Light Band eHMI in Interactions Between Automated Vehicles and PedestriansDebargha Dey, Azra Habibovic, Bastian Pfleging, Marieke H. Martens et al.CHI 2020 · 146 citations
- Light it Up: Evaluating Versatile Autonomous Vehicle-Cyclist External Human-Machine InterfacesAmmar Al-Taie, Graham A. Wilson, Euan Freeman, Frank E. Pollick et al.CHI 2024 · 16 citations
- "It Must Be Gesturing Towards Me": Gesture-Based Interaction between Autonomous Vehicles and PedestriansXiang Chang, Zihe Chen, Xiaoyan Dong, Yuxin Cai et al.CHI 2024 · 14 citations
- Beyond Vacuuming: How Can We Exploit Domestic Robots' Idle Time?Yoshiaki Shiokawa, Winnie Chen, Aditya Shekhar Nittala, Jason Alexander et al.CHI 2025 · 9 citations
Related papers
- Exploring LLMs for Generating Communicational Actions of External Interfaces on Autonomous VehiclesXinyue Gui, Ding Xia, Mark Colley, Stela Hanbyeol Seo et al.UbiComp 2026
- Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous VehiclesJiawei Liu, Xun Gong, Fen Fang, Muli Yang et al.CVPR 2026
- LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model ProgramsYunsheng Ma, Can Cui, Xu Cao, Wenqian Ye et al.CVPR 2024 · 39 citations
- Improving External Communication of Automated Vehicles Using Bayesian OptimizationMark Colley, Pascal Jansen, Mugdha Keskar, Enrico RukzioCHI 2025 · 7 citations
- Exploring Large Language Model-Driven Agents for Environment-Aware Spatial Interactions and Conversations in Virtual Reality Role-Play ScenariosZiming Li, Huadong Zhang, Chao Peng, Roshan L. PeirisIEEE VR 2025 · 18 citations
