Lune

ICLR2025Top-tier venue

Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference

Matthew Riemer, Gopeshh Subbaraj, Glen Berseth, Irina Rish

Abstract

Realtime environments change even as agents perform action inference and learning, thus requiring high interaction frequencies to effectively minimize regret. However, recent advances in machine learning involve larger neural networks with longer inference times, raising questions about their applicability in realtime systems where reaction time is crucial. We present an analysis of lower bounds on regret in realtime reinforcement learning (RL) environments to show that minimizing long-term regret is generally impossible within the typical sequential interaction and learning paradigm, but often becomes possible when sufficient asynchronous compute is available. We propose novel algorithms for staggering asynchronous inference processes to ensure that actions are taken at consistent time intervals, and demonstrate that use of models with high action inference times is only constrained by the environment's effective stochasticity over the inference horizon, and not by action frequency. Our analysis shows that the number of inference processes needed scales linearly with increasing inference times while enabling use of models that are multiple orders of magnitude larger than existing approaches when learning from a realtime simulation of Game Boy games such as Pokémon and Tetris.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Is Pessimism Provably Efficient for Offline RL? cites Conservative Q-Learning for Offline Reinforcement Learning
  • Bigger, Better, Faster: Human-level Atari with human-level efficiency cites Leveraging Procedural Generation to Benchmark Reinforcement Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Handling Delay in Real-Time Reinforcement LearningIvan Anokhin, Rishav Rishav, Matthew Riemer, Stephen Chung et al.ICLR 2025 · 225 citations
  2. Real-Time Reasoning Agents in Evolving EnvironmentsYule Wen, Yixin Ye, Yanzhe Zhang, Diyi Yang et al.ICLR 2026 · 10 citations

Builds on14

  1. Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker, Sergey LevineNeurIPS 2020 · 2,881 citations
  2. Leveraging Procedural Generation to Benchmark Reinforcement LearningKarl Cobbe, Christopher Hesse, Jacob Hilton, John SchulmanICML 2020 · 685 citations
  3. Is Pessimism Provably Efficient for Offline RL?Ying Jin, Zhuoran Yang, Zhaoran WangICML 2021 · 419 citations
  4. Bigger, Better, Faster: Human-level Atari with human-level efficiencyMax Schwarzer, Johan S. Obando-Ceron, Aaron C. Courville, Marc G. Bellemare et al.ICML 2023 · 155 citations
  5. A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement LearningDong-Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun et al.ICML 2021 · 66 citations

Related papers

  1. Pausing Policy Learning in Non-stationary Reinforcement LearningHyunin Lee, Ming Jin, Javad Lavaei, Somayeh SojoudiICML 2024 · 4 citations
  2. Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State ObservationsMinshuo Chen, Yu Bai, H. Vincent Poor, Mengdi WangNeurIPS 2023 · 19 citations
  3. Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement LearningSid Bharthulwar, Stone Tao, Hao SuNeurIPS 2025 · 6 citations
  4. A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics TasksThomas Schmied, Thomas Adler, Vihang Prakash Patil, Maximilian Beck et al.ICML 2025
  5. Breaking the Performance Ceiling in Reinforcement Learning requires Inference StrategiesFélix Chalumeau, Daniel Rajaonarivonivelomanantsoa, Ruan John de Kock, Juan Claude Formanek et al.NeurIPS 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines