FreeAskWorld: An Interactive and Closed-Loop Simulator for Human-Centric Embodied AI
Yuhang Peng, Yizhou Pan, Xinning He, Jihaoyu Yang, Xinyu Yin, Han Wang, Xiaoji Zheng, Chao Gao, Jiangtao Gong
Abstract
As embodied intelligence emerges as a core frontier in artificial intelligence research, simulation platforms must evolve beyond low-level physical interactions to capture complex, human-centered social behaviors. We introduce FreeAskWorld, an interactive simulation framework that integrates large language models (LLMs) for high-level behavior planning and semantically grounded interaction, informed by theories of intention and social cognition. Our framework supports scalable, realistic human-agent simulations and includes a modular data generation pipeline tailored for diverse embodied tasks. To validate the framework, we extend the classic Vision-and-Language Navigation (VLN) task into a interaction enriched Direction Inquiry setting, wherein agents can actively seek and interpret navigational guidance. We present and publicly release FreeAskWorld, a large-scale benchmark dataset comprising reconstructed environments, six diverse task types, 16 core object categories, 63,429 annotated sample frames, and more than 17 hours of interaction data to support training and evaluation of embodied AI systems. We benchmark VLN models, and human participants under both open-loop and closed-loop settings. Experimental results demonstrate that models fine-tuned on FreeAskWorld outperform their original counterparts, achieving enhanced semantic understanding and interaction competency. These findings underscore the efficacy of socially grounded simulation frameworks in advancing embodied AI systems toward sophisticated high-level planning and more naturalistic human-agent interaction. Importantly, our work underscores that interaction itself serves as an additional information modality.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext e205a129-1817-491f-bda6-3f09648aacb0Builds on6
- Scaling Data Generation in Vision-and-Language NavigationZun Wang, Jialu Li, Yicong Hong, Yi Wang et al.ICCV 2023 · 136 citations
- SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and ModelingZhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu et al.ICCV 2023 · 73 citations
- VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language NavigationJialu Li, Aishwarya Padmakumar, Gaurav S. Sukhatme, Mohit BansalAAAI 2024 · 13 citations
- Virtual Community: An Open World for Humans, Robots, and SocietyQinhong Zhou, Hongxin Zhang, Xiangye Lin, Zheyuan Zhang et al.ICLR 2026 · 12 citations
- nuScenes: A Multimodal Dataset for Autonomous DrivingHolger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora et al.CVPR 2020
Related papers
- BOP-ASK: Object-Interaction Reasoning for Vision-Language ModelsVineet Bhat, Sungsu Kim, Valts Blukis, Greg Heinrich et al.CVPR 2026 · 6 citations
- An Embodied Generalist Agent in 3D WorldJiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu et al.ICML 2024 · 361 citations
- ProcWorld: Benchmarking Large Model Planning in Reachability-Constrained EnvironmentsDong Wang, Xinghang Li, Zhengshen Zhang, Jirong Liu et al.EMNLP 2025
- VirtualEnv: A Platform for Embodied AI ResearchKabir Swain, Sijie Han, Ayush Raina, Jin Zhang et al.AAAI 2026
- HIS-GPT: Towards 3D Human-In-Scene Multimodal UnderstandingJiahe Zhao, Ruibing Hou, Zejie Tian, Hong Chang et al.ICCV 2025 · 6 citations
