ETA: Efficiency through Thinking Ahead, a Dual Approach to Self-Driving with Large Models
Shadi Hamdan, Chonghao Sima, Zetong Yang, Hongyang Li, Fatma Güney
Abstract
How can we benefit from large models without sacrificing inference speed, a common dilemma in self-driving systems? A prevalent solution is a dual-system architecture, employing a small model for rapid, reactive decisions and a larger model for slower but more informative analyses. Existing dual-system designs often implement parallel architectures where inference is either directly conducted using the large model at each current frame or retrieved from previously stored inference results. However, these works still struggle to enable large models for a timely response to every online frame. Our key insight is to shift intensive computations of the current frame to previous time steps and perform a batch inference of multiple time steps to make large models respond promptly to each time step. To achieve the shifting, we introduce Efficiency through Thinking Ahead (ETA), an asynchronous system designed to: (1) propagate informative features from the past to the current frame using future predictions from the large model, (2) extract current frame features using a small model for real-time responsiveness, and (3) integrate these dual features via an action mask mechanism that emphasizes action-critical image regions. Evaluated on the Bench2Drive CARLA Leaderboard-v2 benchmark, ETA advances state-of-the-art performance by 8% with a driving score of 69.53 while maintaining a near-real-time inference speed at 50 ms. Code and checkpoints can be found here.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- PaLM-E: An Embodied Multimodal Language Model cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Visual Instruction Tuning cites PaLM-E: An Embodied Multimodal Language Model
- ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving cites Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
- ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving cites VAD: Vectorized Scene Representation for Efficient Autonomous Driving
- DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving cites Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
- DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving cites VAD: Vectorized Scene Representation for Efficient Autonomous Driving
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving cites VAD: Vectorized Scene Representation for Efficient Autonomous Driving
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving cites DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving
Cited by top-tier papers4
- DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous DrivingZhenjie Yang, Yilin Chai, Xiaosong Jia, Qifeng Li et al.CVPR 2026 · 108 citations
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous DrivingPeizheng Li, Zhenghao Zhang, David Holtz, Hang Yu et al.CVPR 2026 · 32 citations
- ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous DrivingZhiyu Zheng, Shaoyu Chen, haoran yin, xinbang zhang et al.CVPR 2026 · 14 citations
- PlannerRFT: Reinforcing Diffusion Planners through Closed-Loop and Sample-Efficient Fine-TuningHongchen Li, Tianyu Li, Jiazhi Yang, Mingyang Shang et al.CVPR 2026 · 13 citations
Builds on17
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch et al.ICML 2023 · 2,601 citations
- VAD: Vectorized Scene Representation for Efficient Autonomous DrivingBo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao et al.ICCV 2023 · 602 citations
- Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong BaselinePenghao Wu, Xiaosong Jia, Li Chen, Junchi Yan et al.NeurIPS 2022 · 444 citations
Related papers
- Unifying Language-Action Understanding and Generation for Autonomous DrivingXinyang Wang, Qian Liu, Wenjie Ding, Zhao Yang et al.CVPR 2026 · 10 citations
- Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous DrivingXiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie et al.CVPR 2023
- DeepSight: Long-Horizon World Modeling via Latent States Prediction for End-to-End Autonomous DrivingLingjun Zhang, Changjie Wu, Linzhe Shi, Jiangyang Li et al.ICML 2026
- ADaPT: Token-Level Decoupling for Efficient Large Reasoning ModelsTingyun Li, Zishang Jiang, Jinyi Han, Xinyi Wang et al.ACL Findings 2026
- AnA: An Attentive Autonomous Driving SystemWonkyo Choe, Rongxiang Wang, Felix Xiaozhu LinASPLOS 2025
