Lune

ICML2026Top-tier venue

Mind the Budget: Accelerating Deep Reinforcement Learning using Constrained Early Exit Neural Networks

Julien Brosseit, Jasper Hoffmann, Joschka Boedecker

Abstract

Early exit neural networks, which adapt computation to input complexity, have proven effective in supervised learning but remain largely unexplored in deep reinforcement learning (DRL). In this paper, we propose Budgeted EXit Actor (BEXA), a novel actor-critic architecture that integrates early exit branches into the actor network. To ensure a mathematically principled trade-off between performance and inference expenditure, the exit decision is formulated as a constrained linear program during training, the solution of which is subsequently amortized to enable highly efficient runtime execution. BEXA is general, governed by an interpretable budget parameter, and compatible with any off-policy actor-critic method. We evaluate BEXA by integrating it with standard off-policy algorithms like SAC and TD3 on a suite of MuJoCo tasks. Our results demonstrate a substantial improvement in inference efficiency with minimal or no loss in performance. These findings highlight early exits as a promising direction for improving computational efficiency in DRL.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self DistillationLinfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen et al.ICCV 2019 · 1,069 citations
  2. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph CompilationJason Ansel, Edward Z. Yang, Horace He, Natalia Gimelshein et al.ASPLOS 2024 · 693 citations
  3. BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley et al.NeurIPS 2020 · 473 citations
  4. Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous controlMichal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Milos et al.NeurIPS 2024 · 119 citations
  5. Stop Regressing: Training Value Functions via Classification for Scalable Deep RLJesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga et al.ICML 2024 · 118 citations

Related papers

  1. Improving DNN Inference Throughput Using Practical, Per-Input Compute AdaptationAnand Padmanabha Iyer, Mingyu Guan, Yinwei Dai, Rui Pan et al.SOSP 2024 · 1 citation
  2. Online Meta-Critic Learning for Off-Policy Actor-Critic MethodsWei Zhou, Yiying Li, Yongxin Yang, Huaimin Wang et al.NeurIPS 2020 · 54 citations
  3. Fast yet Safe: Early-Exiting with Risk ControlMetod Jazbec, Alexander Timans, Tin Hadzi Veljkovic, Kaspar Sakmann et al.NeurIPS 2024 · 35 citations
  4. Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform SamplingChe Wang, Yanqiu Wu, Quan Vuong, Keith W. RossICML 2020 · 38 citations
  5. RVI-SAC: Average Reward Off-Policy Deep Reinforcement LearningYukinari Hisaki, Isao OnoICML 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines