Lune

NeurIPS2024Top-tier venue

Dual Critic Reinforcement Learning under Partial Observability

Jinqiu Li, Enmin Zhao, Tong Wei, Junliang Xing, Shiming Xiang

Abstract

Partial observability in environments poses significant challenges that impede the formation of effective policies in reinforcement learning. Prior research has shown that borrowing the complete state information can enhance sample efficiency. This strategy, however, frequently encounters unstable learning with high variance in practical applications due to the over-reliance on complete information. This paper introduces DCRL, a Dual Critic Reinforcement Learning framework designed to adaptively harness full-state information during training to reduce variance for optimized online performance. In particular, DCRL incorporates two distinct critics: an oracle critic with access to complete state information and a standard critic functioning within the partially observable context. It innovates a synergistic strategy to meld the strengths of the oracle critic for efficiency improvement and the standard critic for variance reduction, featuring a novel mechanism for seamless transition and weighting between them. We theoretically prove that DCRL mitigates the learning variance while maintaining unbiasedness. Extensive experimental analyses across the Box2D and Box3D environments have verified DCRL's superior performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Scalable Policy-Based RL Algorithms for POMDPs cites History Compression via Language Models in Reinforcement Learning
  • History Compression via Language Models in Reinforcement Learning cites Stabilizing Transformers for Reinforcement Learning
  • Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs cites Stabilizing Transformers for Reinforcement Learning
  • Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs cites Variational Recurrent Models for Solving Partially Observable Control Tasks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Scalable Policy-Based RL Algorithms for POMDPsAmeya Anjarlekar, S. Rasoul Etesami, R. SrikantNeurIPS 2025 · 6 citations
  2. Bayesian Ensemble for Sequential Decision-MakingRui Liu, Enmin Zhao, Lu Wang, Yu Li et al.ICLR 2026 · 2 citations
  3. PAMDP: Interact to Persona Alignment via a Partially Observable Markov Decision ProcessZhe Yang, Yi Huang, Si Chen, Xiaoting Wu et al.ICLR 2026

Builds on12

  1. Stabilizing Transformers for Reinforcement LearningEmilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu et al.ICML 2020 · 464 citations
  2. Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPsTianwei Ni, Benjamin Eysenbach, Ruslan SalakhutdinovICML 2022 · 162 citations
  3. Variational Recurrent Models for Solving Partially Observable Control TasksDongqi Han, Kenji Doya, Jun TaniICLR 2020 · 75 citations
  4. Efficient Continuous Control with Double Actors and Regularized CriticsJiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Xiu LiAAAI 2022 · 69 citations
  5. History Compression via Language Models in Reinforcement LearningFabian Paischer, Thomas Adler, Vihang Patil, Angela Bitto-Nemling et al.ICML 2022 · 53 citations

Related papers

  1. Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement LearningHa Manh Bui, Metod Jazbec, Eric Nalisnick, Anqi LiuICML 2026
  2. Learning Belief Representations for Partially Observable Deep RLAndrew Wang, Andrew C. Li, Toryn Q. Klassen, Rodrigo Toro Icarte et al.ICML 2023 · 21 citations
  3. Provable Partially Observable Reinforcement Learning with Privileged InformationYang Cai, Xiangyu Liu, Argyris Oikonomou, Kaiqing ZhangNeurIPS 2024 · 22 citations
  4. CTD4 - a Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple CriticsDavid Valencia, Henry Williams, Yuning Xing, Trevor Gee et al.AAAI 2025 · 6 citations
  5. Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State AccessDaniel Ebi, Damien Ernst, Klemens Böhm, Gaspard LambrechtsICML 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines