Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span
Heeseung Yun, Joonil Na, Jaeyeon Kim, Calvin Murdock, Gunhee Kim
Abstract
People continuously perceive and interact with their surroundings based on underlying intentions that drive their exploration and behaviors. While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies. We address the challenge of egocentric 3D visual span forecasting, predicting where a person's visual perception will focus next within their three-dimensional environment. To this end, we propose EgoSpanLift, a novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes. EgoSpanLift converts SLAM-derived keypoints into gaze-compatible geometry and extracts volumetric visual span regions. We further combine EgoSpanLift with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid. In addition, we curate a comprehensive benchmark from raw egocentric multisensory data, creating a testbed with 364.6K samples for 3D visual span forecasting. Our approach outperforms competitive baselines for egocentric 2D gaze anticipation and 3D localization while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 32843b14-5066-42e4-830d-05564d40394aCited by top-tier papers1
Ask how each one uses itBuilds on23
- Ego4D: Around the World in 3, 000 Hours of Egocentric VideoKristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis et al.CVPR 2022 · 525 citations
- OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy PredictionYunpeng Zhang, Zheng Zhu, Dalong DuICCV 2023 · 354 citations
- H2O: Two Hands Manipulating Objects for First Person Interaction RecognitionTaein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo et al.ICCV 2021 · 271 citations
- What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality AttentionAntonino Furnari, Giovanni Maria FarinellaICCV 2019 · 204 citations
- Ego-Pose Estimation and Forecasting As Real-Time PD ControlYe Yuan, Kris KitaniICCV 2019 · 147 citations
Related papers
- Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory ForecastingWentao Bao, Lele Chen, Libing Zeng, Zhong Li et al.ICCV 2023 · 34 citations
- COPILOT: Human-Environment Collision Prediction and Localization from Egocentric VideosBoxiao Pan, Bokui Shen, Davis Rempe, Despoina Paschalidou et al.ICCV 2023 · 3 citations
- LAMP: Localization Aware Multi-camera People Tracking in Metric 3D WorldNan Yang, Julian Straub, Fan Zhang, Richard A. Newcombe et al.CVPR 2026 · 1 citation
- UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and GenerationChaitanya Patel, Hiroki Nakamura, Yuta Kyuragi, Kazuki Kozuka et al.ICCV 2025 · 3 citations
- Towards Visual Query Localization in the 3D WorldLiang Peng, Bohan Tan, Zhipeng Zhang, Haobo Li et al.CVPR 2026 · 1 citation
