STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
Hossein Goli, Michael Gimelfarb, Nathan de Lara, Haruki Nishimura, Masha Itkina, Florian Shkurti
Abstract
Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents overregularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods †
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Planning with Diffusion for Flexible Behavior Synthesis cites Offline Reinforcement Learning as One Big Sequence Modeling Problem
- Planning with Diffusion for Flexible Behavior Synthesis cites MOPO: Model-based Offline Policy Optimization
- Planning with Diffusion for Flexible Behavior Synthesis cites Diffusion Models Beat GANs on Image Synthesis
- Planning with Diffusion for Flexible Behavior Synthesis cites Denoising Diffusion Probabilistic Models
- Generative View Stitching cites Denoising Diffusion Probabilistic Models
- Offline Reinforcement Learning as One Big Sequence Modeling Problem cites MOPO: Model-based Offline Policy Optimization
- Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers1
Builds on20
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
- Planning with Diffusion for Flexible Behavior SynthesisMichael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey LevineICML 2022 · 1,115 citations
- MOPO: Model-based Offline Policy OptimizationTianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon et al.NeurIPS 2020 · 989 citations
- Offline Reinforcement Learning as One Big Sequence Modeling ProblemMichael Janner, Qiyang Li, Sergey LevineNeurIPS 2021 · 950 citations
Related papers
- State-Covering Trajectory Stitching for Diffusion PlannersKyowoon Lee, Jaesik ChoiNeurIPS 2025 · 17 citations
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory StitchingGuanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long et al.ICML 2024 · 41 citations
- DIDI: Diffusion-Guided Diversity for Offline Behavioral GenerationJinxin Liu, Xinghong Guo, Zifeng Zhuang, Donglin WangICML 2024 · 3 citations
- Distributional Offline Policy Evaluation with Predictive Error GuaranteesRunzhe Wu, Masatoshi Uehara, Wen SunICML 2023 · 19 citations
- Generative Trajectory Stitching through Diffusion CompositionYunhao Luo, Utkarsh A. Mishra, Yilun Du, Danfei XuNeurIPS 2025 · 48 citations
