PANDALens: Towards AI-Assisted In-Context Writing on OHMD During Travels
Runze Cai, Nuwan Janaka, Yang Chen, Lucia J. Wang, Shengdong Zhao, Can Liu
Abstract
While effective for recording and sharing experiences, traditional in-context writing tools are relatively passive and unintelligent, serving more like instruments rather than companions. This reduces primary task (e.g., travel) enjoyment and hinders high-quality writing. Through formative study and iterative development, we introduce PANDALens, a Proactive AI Narrative Documentation Assistant built on an Optical See-Through Head Mounted Display that supports personalized documentation in everyday activities. PANDALens observes multimodal contextual information from user behaviors and environment to confirm interests and elicit contemplation, and employs Large Language Models to transform such multimodal information into coherent narratives with significantly reduced user effort. A real-world travel scenario comparing PANDALens with a smartphone alternative confirmed its effectiveness in improving writing quality and travel enjoyment while minimizing user effort. Accordingly, we propose design guidelines for AI-assisted in-context writing, highlighting the potential of transforming them from tools to intelligent companions.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 6b674773-151e-46b4-a0d1-1d66a1ac7eafCited by top-tier papers12
- Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System DesignYongquan 'Owen' Hu, Jingyu Tang, Xinya Gong, Zhongyi Zhou et al.CHI 2025 · 37 citations
- Amuse: Human-AI Collaborative Songwriting with Multimodal InspirationsYewon Kim, Sung-Ju Lee, Chris DonahueCHI 2025 · 35 citations
- How CO2STLY Is CHI? The Carbon Footprint of Generative AI in HCI Research and What We Should Do About ItNanna Inie, Jeanette Falk, Raghavendra SelvanCHI 2025 · 33 citations
- Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing ProcessMohi Reza, Jeb Thomas-Mitchell, Peter Dushniku, Nathan Laundry et al.CSCW 2025 · 29 citations
- AiGet: Transforming Everyday Moments into Hidden Knowledge Discovery with AI Assistance on Smart GlassesRunze Cai, Nuwan Janaka, Hyeongcheol Kim, Yang Chen et al.CHI 2025 · 26 citations
Builds on20
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch et al.ICML 2023 · 2,601 citations
- Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM PromptsJ. D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, Qian YangCHI 2023 · 892 citations
- Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to DesignQian Yang, Aaron Steinfeld, Carolyn P. Rosé, John ZimmermanCHI 2020 · 604 citations
Related papers
- How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision PeopleRicardo E. Gonzalez Penuela, Crescentia Jung, Sharon Y. Lin, Ruiying Hu et al.CHI 2026 · 1 citation
- MIND: Empowering Mental Health Clinicians with Multimodal Data Insights through a Narrative DashboardRuishi Zou, Shiyu Xu, Margaret E. Morris, Jihan Ryu et al.CHI 2026 · 2 citations
- Pro 2 Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural TasksLilin Xu, Bufang Yang, Siyang Jiang, Kaiwei Liu et al.UbiComp 2026
- Proactive AI as a Catalyst for Creativity? Balancing Human Agency and AI Contribution in Collaborative Story WritingYiwen Yin, Mingze Wu, Ruijie Huang, Xin Tong et al.CHI 2026 · 1 citation
- PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XREsen K. Tütüncü, Qian Zhou, Frederik Brudy, George W. Fitzmaurice et al.CHI 2026 · 2 citations
