Lune

ICML2026Top-tier venue

Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning

Thanh Nguyen, Tri Ton, Hongbin Choe, Minh-Tung Luu, Chang D. Yoo

Abstract

Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation typically introduce auxiliary networks, policy distillation, or multi-phase training, which frequently compromise simplicity, stability, or performance. To address these limitations, we introduce Bootstrapped Flow Q-Learning (BFQ), a novel framework that enables accurate single-step action generation during both training and inference—without auxiliary networks or distillation procedures. BFQ adopts a divide-and-conquer view of the displacement vector along the flow path: it begins by learning short-range displacements that can be accurately estimated from the Flow Matching marginal velocity, and bootstraps these components to directly learn a noise-to-action mapping in a single step. This formulation eliminates multi-step denoising, resulting in a learning procedure that is substantially faster, simpler, and more robust. Extensive D4RL evaluations show that BFQ improves performance while significantly reducing computational cost compared to multi-step diffusion baselines, demonstrating that single-step action generation suffices for high-performance offline Reinforcement Learning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Offline Reinforcement Learning with Implicit Q-Learning cites Conservative Q-Learning for Offline Reinforcement Learning
  • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites Denoising Diffusion Probabilistic Models
  • Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning cites Offline Reinforcement Learning with Implicit Q-Learning
  • Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement LearningMinh-Tung Luu, Hwanhee Kim, Younghwan Lee, Chang D. YooICML 2026 · 1 citation

Builds on26

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari et al.ICML 2024 · 3,620 citations
  4. Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker, Sergey LevineNeurIPS 2020 · 2,881 citations
  5. Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey LevineICLR 2022 · 1,402 citations

Related papers

  1. One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement LearningXuan Thanh Nguyen, Chang Dong YooICLR 2026 · 11 citations
  2. One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlowZeyuan Wang, Da Li, Yulin Chen, Ye Shi et al.AAAI 2026 · 6 citations
  3. Direct Flow Q-LearningShicheng Cao, Jingrui Jia, Wenyu Li, Feng Duan et al.ICML 2026
  4. Diffusion Policies Creating a Trust Region for Offline Reinforcement LearningTianyu Chen, Zhendong Wang, Mingyuan ZhouNeurIPS 2024 · 51 citations
  5. Flow Q-LearningSeohong Park, Qiyang Li, Sergey LevineICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines