CVPR Findings2026
Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
Chihiro Noguchi, Takaki Yamamoto
Abstract
End-to-end (E2E) autonomous driving models that take only camera images as input and directly predict a future trajectory are appealing for their computational efficiency and potential for improved generalization via unified optimization; however, persistent failure modes remain due to reliance on imitation learning (IL). While online reinforcement learning (RL) could mitigate IL-induced issues, the computational burden of neural rendering-based simulation and large E2E networks renders iterative reward and hyperparameter tuning costly. We introduce a cameraonly E2E offline RL framework that performs no additional exploration and trains solely on a fixed simulator dataset. Offline RL offers strong data efficiency and rapid experimental iteration, yet is susceptible to instability from overestimation on out-of-distribution (OOD) actions. To address this, we construct pseudo ground-truth trajectories from expert driving logs and use them as a behavior regularization signal, suppressing imitation of unsafe or suboptimal behavior while stabilizing value learning. Training and closed-loop evaluation are conducted in a neural rendering environment learned from the public nuScenes dataset. Empirically, the proposed method achieves substantial improvements in collision rate and route completion compared with IL baselines. Our code is available at https://github.com/ToyotaInfoTech/PEBC.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Offline Reinforcement Learning with Implicit Q-Learning cites Decision Transformer: Reinforcement Learning via Sequence Modeling
- Offline Reinforcement Learning with Implicit Q-Learning cites Conservative Q-Learning for Offline Reinforcement Learning
- Decision Transformer: Reinforcement Learning via Sequence Modeling cites Conservative Q-Learning for Offline Reinforcement Learning
Builds on32
- Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
- 3D Gaussian Splatting for Real-Time Radiance Field RenderingBernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George DrettakisSIGGRAPH 2023 · 5,687 citations
- Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker, Sergey LevineNeurIPS 2020 · 2,881 citations
- Decision Transformer: Reinforcement Learning via Sequence ModelingLili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee et al.NeurIPS 2021 · 2,557 citations
- Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey LevineICLR 2022 · 1,402 citations
Related papers
- CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous DrivingXiaoji Zheng, Ziyuan Yang, Yanhao Chen, Yuhang PENG et al.ICML 2026
- End-to-End Urban Driving by Imitating a Reinforcement Learning CoachZhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu et al.ICCV 2021 · 313 citations
- DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning ModelRui Yu, Xianghang Zhang, Runkai Zhao, Huaicheng Yan et al.ICCV 2025 · 19 citations
- DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous DrivingShuyao Shang, Yuntao Chen, Yuqi Wang, Yingyan Li et al.NeurIPS 2025 · 49 citations
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement LearningHao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao et al.NeurIPS 2025 · 92 citations
