Lune

CHI2025Top-tier venue

GazeNoter: Co-Piloted AR Note-Taking via Gaze Selection of LLM Suggestions to Match Users' Intentions

Hsin-Ruey Tsai, Shih-Kang Chiu, Bryan Wang

Abstract

Note-taking is critical during speeches and discussions, serving not only for later summarization and organization but also for real-time question and opinion reminding in question-and-answer sessions or timely contributions in discussions. Manually typing on smartphones for note-taking could be distracting and increase cognitive load for users. While large language models (LLMs) are used to automatically generate summaries and highlights, the content generated by artificial intelligence (AI) may not match users' intentions without user input or interaction. Therefore, we propose an AI-copiloted augmented reality (AR) system, GazeNoter, to allow users to swiftly select diverse LLM-generated suggestions via gaze on an AR headset for real-time note-taking. GazeNoter leverages an AR headset as a medium for users to swiftly adjust the LLM output to match their intentions, forming a user-in-the-loop AI system for both within-context and beyond-context notes. We conducted two user studies to verify the usability of GazeNoter in attending speeches in a static sitting condition and walking meetings and discussions in a mobile walking condition, respectively.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals cites AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts
  • Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals cites TaleBrush: Sketching Stories with Generative Pretrained Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System DesignYongquan 'Owen' Hu, Jingyu Tang, Xinya Gong, Zhongyi Zhou et al.CHI 2025 · 37 citations
  2. AttentiveLearn: Personalized Post-Lecture Support for Gaze-Aware Immersive LearningShi Liu, Martin Feick, Linus Bierhoff, Alexander MaedcheCHI 2026 · 2 citations
  3. AIDED: Augmenting Interior Design with Human Experience Data for Designer-AI Co-DesignYang Chen Lin, Chen-Ying Chien, Kai-Hsin Hou, Hung-Yu Chen et al.CHI 2026 · 2 citations
  4. Seeing Eye to Eye: Enabling Cognitive Alignment Through Shared First-Person Perspective in Human-AI Collaboration: Seeing Eye to EyeZhuyu Teng, Pei Chen, Yichen Cai, Ruoqing Lu et al.CHI 2026 · 2 citations

Builds on37

  1. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model PromptsTongshuang Wu, Michael Terry, Carrie Jun CaiCHI 2022 · 465 citations
  2. Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry ProfessionalsPiotr Mirowski, Kory W. Mathewson, Jaylen Pittman, Richard EvansCHI 2023 · 235 citations
  3. TaleBrush: Sketching Stories with Generative Pretrained Language ModelsJohn Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee et al.CHI 2022 · 202 citations
  4. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language ModelsStephen Brade, Bryan Wang, Maurício Sousa, Sageev Oore et al.UIST 2023 · 179 citations
  5. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language ModelsSangho Suh, Bryan Min, Srishti Palani, Haijun XiaUIST 2023 · 147 citations

Related papers

  1. Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision ModelsAda Yi Zhao, Aditya Gunturu, Ellen Yi-Luen Do, Ryo SuzukiUIST 2025 · 12 citations
  2. AI-Powered Conversational Assistance in Augmented Reality for Multi-Step TasksJuliana H. Madritsch, Tomislav Duricic, Neven A. M. ElSayed, Simone Kopeinik et al.IEEE VR 2026 · 1 citation
  3. ARTiST: Automated Text Simplification for Task Guidance in Augmented RealityGuande Wu, Jing Qian, Sonia Castelo Quispe, Shaoyu Chen et al.CHI 2024 · 20 citations
  4. SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live InteractionsBufang Yang, Yunqi Guo, Lilin Xu, Zhenyu Yan et al.UbiComp 2025 · 26 citations
  5. "The less I type, the better": How AI Language Models can Enhance or Impede Communication for AAC UsersStephanie Valencia, Richard Cave, Krystal Kallarackal, Katie Seaver et al.CHI 2023 · 92 citations
Dusk over the sea between two cliffs drawn in fine vertical lines