Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Haoran He, Emmanuel Bengio, Qingpeng Cai, Ling Pan
Abstract
The Generative Flow Network (GFlowNet) is a probabilistic framework in which an agent learns a stochastic policy and flow functions to sample objects proportionally to an unnormalized reward function. A number of recent works explored connections between GFlowNets and maximum entropy (MaxEnt) RL, which modifies the standard objective of RL agents by learning an entropy-regularized objective. However, the relationship between GFlowNets and standard RL remains largely unexplored, despite the inherent similarities in their sequential decisionmaking nature. While GFlowNets can discover diverse solutions through specialized flow-matching objectives, connecting them can simplify their implementation through established RL principles and improve RL's diverse solution discovery capabilities. In this paper, we bridge this gap by revealing a fundamental connection between GFlowNets and one RL's most basic components -policy evaluation. Surprisingly, we find that the value function obtained from evaluating a uniform policy is closely associated with the flow functions in GFlowNets through the lens of flow iteration under certain structural conditions. Building upon these insights, we introduce a rectified random policy evaluation (RPE) algorithm, which achieves the same reward-matching effect as GFlowNets based on simply evaluating a fixed random policy in these cases, offering a new perspective. Empirical results across extensive benchmarks demonstrate that RPE achieves competitive results compared to previous approaches, shedding light on the previously overlooked connection between (non-MaxEnt) RL and GFlowNets.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Evaluating GFlowNet from partial episodes for stable and flexible policy-based training cites Learning GFlowNets From Partial Episodes For Improved Convergence And Stability
- Evaluating GFlowNet from partial episodes for stable and flexible policy-based training cites Trajectory balance: Improved credit assignment in GFlowNets
- Evaluating GFlowNet from partial episodes for stable and flexible policy-based training cites Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
- Revisiting Non-Acyclic GFlowNets in Discrete Environments cites Biological Sequence Design with GFlowNets
- Revisiting Non-Acyclic GFlowNets in Discrete Environments cites Learning GFlowNets From Partial Episodes For Improved Convergence And Stability
- Revisiting Non-Acyclic GFlowNets in Discrete Environments cites Trajectory balance: Improved credit assignment in GFlowNets
- Revisiting Non-Acyclic GFlowNets in Discrete Environments cites Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
- Biological Sequence Design with GFlowNets cites Trajectory balance: Improved credit assignment in GFlowNets
- Biological Sequence Design with GFlowNets cites Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
- Learning GFlowNets From Partial Episodes For Improved Convergence And Stability cites Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
- Trajectory balance: Improved credit assignment in GFlowNets cites Learning GFlowNets From Partial Episodes For Improved Convergence And Stability
- Trajectory balance: Improved credit assignment in GFlowNets cites Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
- Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards cites Softmax Deep Double Deterministic Policy Gradients
- Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards cites Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
Cited by top-tier papers3
- Random Policy Valuation is Enough for LLM Reasoning with Verifiable RewardsHaoran He, Yuxiao Ye, Qingpeng Cai, Chen Hu et al.ICLR 2026 · 9 citations
- Evaluating GFlowNet from partial episodes for stable and flexible policy-based trainingPuhua Niu, Shili Wu, Xiaoning QianICLR 2026 · 2 citations
- Revisiting Non-Acyclic GFlowNets in Discrete EnvironmentsNikita Morozov, Ian Maksimov, Daniil Tiapkin, Sergey SamsonovICML 2025
Builds on22
- Flow Network based Generative Models for Non-Iterative Diverse Candidate GenerationEmmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup et al.NeurIPS 2021 · 565 citations
- Trajectory balance: Improved credit assignment in GFlowNetsNikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun et al.NeurIPS 2022 · 316 citations
- Biological Sequence Design with GFlowNetsMoksh Jain, Emmanuel Bengio, Alex Hernández-García, Jarrid Rector-Brooks et al.ICML 2022 · 224 citations
- Softmax Deep Double Deterministic Policy GradientsLing Pan, Qingpeng Cai, Longbo HuangNeurIPS 2020 · 138 citations
- Learning GFlowNets From Partial Episodes For Improved Convergence And StabilityKanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio et al.ICML 2023 · 138 citations
Related papers
- Optimizing Backward Policies in GFlowNets via Trajectory Likelihood MaximizationTimofei Gritsaev, Nikita Morozov, Sergey Samsonov, Daniil TiapkinICLR 2025
- Pessimistic Backward Policy for GFlowNetsHyosoon Jang, Yunhui Jang, Minsu Kim, Jinkyoo Park et al.NeurIPS 2024 · 14 citations
- Generative Augmented Flow NetworksLing Pan, Dinghuai Zhang, Aaron C. Courville, Longbo Huang et al.ICLR 2023 · 2 citations
- Towards Understanding and Improving GFlowNet TrainingMax W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas et al.ICML 2023 · 81 citations
- CFlowNets: Continuous Control with Generative Flow NetworksYinchuan Li, Shuang Luo, Haozhi Wang, Jianye HaoICLR 2023 · 6 citations
