RTDiff: Reverse Trajectory Synthesis via Diffusion for Offline Reinforcement Learning
Qianlan Yang, Yu-Xiong Wang
Abstract
In offline reinforcement learning (RL), managing the distribution shift between the learned policy and the static offline dataset is a persistent challenge that can result in overestimated values and suboptimal policies. Traditional offline RL methods address this by introducing conservative biases that limit exploration to wellunderstood regions, but they often overly restrict the agent's generalization capabilities. Recent work has sought to generate trajectories using generative models to augment the offline dataset, yet these methods still struggle with overestimating synthesized data, especially when out-of-distribution samples are produced. To overcome this issue, we propose RTDiff, a novel diffusion-based data augmentation technique that synthesizes trajectories in reverse, moving from unknown to known states. Such reverse generation naturally mitigates the risk of overestimation by ensuring that the agent avoids planning through unknown states. Additionally, reverse trajectory synthesis allows us to generate longer, more informative trajectories that take full advantage of diffusion models' generative strengths while ensuring reliability. We further enhance RTDiff by introducing flexible trajectory length control and improving the efficiency of the generation process through noise management. Our empirical results show that RTDiff significantly improves the performance of several state-of-the-art offline RL algorithms across diverse environments, achieving consistent and superior results by effectively overcoming distribution shift. Our code can be found at https://yanqval.github.io/RTDiff.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- A Minimalist Approach to Offline Reinforcement Learning cites Decision Transformer: Reinforcement Learning via Sequence Modeling
- A Minimalist Approach to Offline Reinforcement Learning cites Conservative Q-Learning for Offline Reinforcement Learning
- Multi-agent Coordination via Flow Matching cites A Minimalist Approach to Offline Reinforcement Learning
- Multi-agent Coordination via Flow Matching cites Offline Reinforcement Learning with Implicit Q-Learning
- Multi-agent Coordination via Flow Matching cites Conservative Q-Learning for Offline Reinforcement Learning
- Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategies cites Offline Reinforcement Learning with Implicit Q-Learning
- Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategies cites Conservative Q-Learning for Offline Reinforcement Learning
- Episodic Memory-Guided Controllable Experience Synthesis for Reinforcement Learning cites A Minimalist Approach to Offline Reinforcement Learning
- Episodic Memory-Guided Controllable Experience Synthesis for Reinforcement Learning cites Offline Reinforcement Learning with Implicit Q-Learning
- Episodic Memory-Guided Controllable Experience Synthesis for Reinforcement Learning cites Elucidating the Design Space of Diffusion-Based Generative Models
- Offline Reinforcement Learning with Implicit Q-Learning cites A Minimalist Approach to Offline Reinforcement Learning
- Offline Reinforcement Learning with Implicit Q-Learning cites Decision Transformer: Reinforcement Learning via Sequence Modeling
- Offline Reinforcement Learning with Implicit Q-Learning cites Conservative Q-Learning for Offline Reinforcement Learning
- Decision Transformer: Reinforcement Learning via Sequence Modeling cites Conservative Q-Learning for Offline Reinforcement Learning
- Reliability-Guaranteed and Reward-Seeking Sequence Modeling for Model-Based Offline Reinforcement Learning cites Offline Reinforcement Learning with Implicit Q-Learning
- Reliability-Guaranteed and Reward-Seeking Sequence Modeling for Model-Based Offline Reinforcement Learning cites Decision Transformer: Reinforcement Learning via Sequence Modeling
- Reliability-Guaranteed and Reward-Seeking Sequence Modeling for Model-Based Offline Reinforcement Learning cites Conservative Q-Learning for Offline Reinforcement Learning
Cited by top-tier papers4
- Multi-agent Coordination via Flow MatchingDongsu Lee, Daehee Lee, Amy ZhangICLR 2026 · 9 citations
- Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment StrategiesDong-Hee Shin, Deok-Joong Lee, Young-Han Son, Tae-Eui KamAAAI 2026 · 3 citations
- Episodic Memory-Guided Controllable Experience Synthesis for Reinforcement LearningXiao Ma, Tian Li, Wu-Jun LiICML 2026
- Reliability-Guaranteed and Reward-Seeking Sequence Modeling for Model-Based Offline Reinforcement LearningShenghong He, Chao Yu, Qian Lin, Yile Liang et al.AAAI 2026
Builds on22
- Elucidating the Design Space of Diffusion-Based Generative ModelsTero Karras, Miika Aittala, Timo Aila, Samuli LaineNeurIPS 2022 · 3,959 citations
- Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker, Sergey LevineNeurIPS 2020 · 2,881 citations
- Decision Transformer: Reinforcement Learning via Sequence ModelingLili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee et al.NeurIPS 2021 · 2,557 citations
- Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey LevineICLR 2022 · 1,402 citations
- A Minimalist Approach to Offline Reinforcement LearningScott Fujimoto, Shixiang Shane GuNeurIPS 2021 · 1,292 citations
Related papers
- Trajectory Generation with Conservative Value Guidance for Offline Reinforcement LearningTieru Wang, Kunbao Wu, Guoshun NanICLR 2026
- BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement LearningYunpeng Qing, Yixiao Chi, Shuo Chen, Shunyu Liu et al.ICML 2026 · 4 citations
- ATraDiff: Accelerating Online Reinforcement Learning with Imaginary TrajectoriesQianlan Yang, Yu-Xiong WangICML 2024 · 2 citations
- GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement LearningJaewoo Lee, Sujin Yun, Taeyoung Yun, Jinkyoo ParkNeurIPS 2024 · 35 citations
- Offline Reinforcement Learning with Reverse Model-based ImaginationJianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu et al.NeurIPS 2021 · 74 citations
