Efficient Multi-agent Offline Coordination via Diffusion-based Trajectory Stitching
Lei Yuan, Yuqi Bian, Lihe Li, Ziqian Zhang, Cong Guan, Yang Yu
Abstract
Learning from offline data without interacting with the environment is a promising way to fully leverage the intelligent decision-making capabilities of multiagent reinforcement learning (MARL). Previous approaches have primarily focused on developing learning techniques, such as conservative methods tailored to MARL using limited offline data. However, these methods often overlook the temporal relationships across different timesteps and spatial relationships between teammates, resulting in low learning efficiency in imbalanced data scenarios. To comprehensively explore the data structure of MARL and enhance learning efficiency, we propose Multi-Agent offline coordination via Diffusionbased Trajectory Stitching (MADiTS), a novel diffusion-based data augmentation pipeline that systematically generates trajectories by stitching high-quality coordination segments together. MADiTS first generates trajectory segments using a trained diffusion model, followed by applying a bidirectional dynamics constraint to ensure that the trajectories align with environmental dynamics. Additionally, we develop an offline credit assignment technique to identify and optimize the behavior of underperforming agents in the generated segments. This iterative procedure continues until a satisfactory augmented episode trajectory is generated within the predefined limit or is discarded otherwise. Empirical results on imbalanced datasets of multiple benchmarks demonstrate that MADiTS significantly improves MARL performance.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective cites Dream to Control: Learning Behaviors by Latent Imagination
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective cites Diffusion Models Beat GANs on Image Synthesis
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective cites Denoising Diffusion Probabilistic Models
- Zero-Shot Trajectory Planning for Signal Temporal Logic Tasks cites Denoising Diffusion Probabilistic Models
- Decision Transformer: Reinforcement Learning via Sequence Modeling cites Conservative Q-Learning for Offline Reinforcement Learning
- Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers2
Builds on40
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
- Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker, Sergey LevineNeurIPS 2020 · 2,881 citations
- Decision Transformer: Reinforcement Learning via Sequence ModelingLili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee et al.NeurIPS 2021 · 2,557 citations
- Dream to Control: Learning Behaviors by Latent ImaginationDanijar Hafner, Timothy P. Lillicrap, Jimmy Ba, Mohammad NorouziICLR 2020 · 1,852 citations
Related papers
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory StitchingGuanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long et al.ICML 2024 · 41 citations
- MADiff: Offline Multi-agent Learning with Diffusion ModelsZhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang et al.NeurIPS 2024 · 116 citations
- State-Covering Trajectory Stitching for Diffusion PlannersKyowoon Lee, Jaesik ChoiNeurIPS 2025 · 17 citations
- RTDiff: Reverse Trajectory Synthesis via Diffusion for Offline Reinforcement LearningQianlan Yang, Yu-Xiong WangICLR 2025
- Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment StrategiesDong-Hee Shin, Deok-Joong Lee, Young-Han Son, Tae-Eui KamAAAI 2026 · 3 citations
