Lune

NeurIPS2025Top-tier venue

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

Hossein Goli, Michael Gimelfarb, Nathan de Lara, Haruki Nishimura, Masha Itkina, Florian Shkurti

Abstract

Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents overregularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods †

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Planning with Diffusion for Flexible Behavior Synthesis cites Offline Reinforcement Learning as One Big Sequence Modeling Problem
  • Planning with Diffusion for Flexible Behavior Synthesis cites MOPO: Model-based Offline Policy Optimization
  • Planning with Diffusion for Flexible Behavior Synthesis cites Diffusion Models Beat GANs on Image Synthesis
  • Planning with Diffusion for Flexible Behavior Synthesis cites Denoising Diffusion Probabilistic Models
  • Generative View Stitching cites Denoising Diffusion Probabilistic Models
  • Offline Reinforcement Learning as One Big Sequence Modeling Problem cites MOPO: Model-based Offline Policy Optimization
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Generative View StitchingChonghyuk Song, Michal Stary, Boyuan Chen, George Kopanas et al.ICLR 2026 · 3 citations

Builds on20

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  3. Planning with Diffusion for Flexible Behavior SynthesisMichael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey LevineICML 2022 · 1,115 citations
  4. MOPO: Model-based Offline Policy OptimizationTianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon et al.NeurIPS 2020 · 989 citations
  5. Offline Reinforcement Learning as One Big Sequence Modeling ProblemMichael Janner, Qiyang Li, Sergey LevineNeurIPS 2021 · 950 citations

Related papers

  1. State-Covering Trajectory Stitching for Diffusion PlannersKyowoon Lee, Jaesik ChoiNeurIPS 2025 · 17 citations
  2. DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory StitchingGuanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long et al.ICML 2024 · 41 citations
  3. DIDI: Diffusion-Guided Diversity for Offline Behavioral GenerationJinxin Liu, Xinghong Guo, Zifeng Zhuang, Donglin WangICML 2024 · 3 citations
  4. Distributional Offline Policy Evaluation with Predictive Error GuaranteesRunzhe Wu, Masatoshi Uehara, Wen SunICML 2023 · 19 citations
  5. Generative Trajectory Stitching through Diffusion CompositionYunhao Luo, Utkarsh A. Mishra, Yilun Du, Danfei XuNeurIPS 2025 · 48 citations
Dusk over the sea between two cliffs drawn in fine vertical lines