Lune

NeurIPS2025Top-tier venue

ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning

Tonghe Zhang, Chao Yu, Sichang Su, Yu Wang

Abstract

We propose ReinFlow, a simple yet effective online reinforcement learning (RL) framework that fine-tunes a family of flow matching policies for continuous robotic control. Derived from rigorous RL theory, ReinFlow injects learnable noise into a flow policy's deterministic path, converting the flow into a discrete-time Markov Process for exact and straightforward likelihood computation. This conversion facilitates exploration and ensures training stability, enabling ReinFlow to fine-tune diverse flow model variants stably, including Rectified Flow [34] and Shortcut Models [18], particularly at very few or even one denoising step. We benchmark ReinFlow in representative locomotion and manipulation tasks, including longhorizon planning with visual input and sparse reward. The episode reward of Rectified Flow policies obtained an average net growth of 135.36% after fine-tuning in challenging legged locomotion tasks while saving denoising steps and 82.63% of wall time compared to state-of-the-art diffusion RL fine-tuning method DPPO [42]. The success rate of the Shortcut Model policies in state and visual manipulation tasks achieved an average net increase of 40.34% after fine-tuning with ReinFlow at four or even one denoising step, whose performance is comparable to fine-tuned DDIM policies while saving computation time for an average of 23.20% . Code, model, and checkpoints available on the project website: https://reinflow.github.io/

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

13 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Flow-GRPO: Training Flow Matching Models via Online RL cites Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  • D²PPO: Diffusion Policy Policy Optimization with Dispersive Loss cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Data Quality in Imitation Learning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Generative Online Reinforcement Learning cites SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling
  • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling cites Data Quality in Imitation Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers13

  1. SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential ModelingYixian Zhang, Shu'ang Yu, Tonghe Zhang, Mo Guang et al.ICLR 2026 · 33 citations
  2. RFS: Reinforcement learning with Residual flow steering for dexterous manipulationEntong Su, Tyler Westenbroek, Anusha Nagabandi, Abhishek GuptaICLR 2026 · 13 citations
  3. D²PPO: Diffusion Policy Policy Optimization with Dispersive LossGuowei Zou, Weibing Li, Hejun Wu, Yukun Qian et al.AAAI 2026 · 3 citations
  4. Generative Online Reinforcement LearningChubin Zhang, Zhenglin Wan, Feng Chen, Fuchao Yang et al.ICML 2026 · 1 citation
  5. Efficient Hybrid SE(3)-Equivariant Visuomotor Flow Policy via Spherical Harmonics for Robot ManipulationQinglun Zhang, Shen Cheng, Tian Dan, Haoqiang Fan et al.CVPR 2026 · 1 citation

Builds on17

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari et al.ICML 2024 · 3,620 citations
  3. Planning with Diffusion for Flexible Behavior SynthesisMichael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey LevineICML 2022 · 1,115 citations
  4. Flow-GRPO: Training Flow Matching Models via Online RLJie Liu, Gongye Liu, Jiajun Liang, Yangguang Li et al.NeurIPS 2025 · 647 citations
  5. Data Quality in Imitation LearningSuneel Belkhale, Yuchen Cui, Dorsa SadighNeurIPS 2023 · 135 citations

Related papers

  1. Mean Flow Policy OptimizationXiaoyi Dong, Xi Zhang, Jian ChengICML 2026
  2. Flow Matching with Injected Noise for Offline-to-Online Reinforcement LearningYongjae Shin, Jongseong Chae, Jongeui Park, Youngchul SungICLR 2026 · 1 citation
  3. One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement LearningXuan Thanh Nguyen, Chang Dong YooICLR 2026 · 11 citations
  4. Flow Matching Policy GradientsDavid McAllister, Songwei Ge, Brent Yi, Chung Min Kim et al.ICLR 2026 · 103 citations
  5. Flow Q-LearningSeohong Park, Qiyang Li, Sergey LevineICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines