Lune

NeurIPS2020Top-tier venue

R-learning in actor-critic model offers a biologically relevant mechanism for sequential decision-making

Sergey Shuvaev, Sarah Starosta, Duda Kvitsiani, Ádám Kepecs, Alexei A. Koulakov

Abstract

When should you continue with your ongoing plans and when should you instead decide to pursue better opportunities? We show in theory and experiment that such stay-or-leave decisions are consistent with deep R-learning both behaviorally and neuronally. Our results suggest that real-world agents leave depleting resources when their reward rate falls below its exponential average, which, we argue, is a Bayes optimal rule in dynamic natural environments. Our work links reinforcement learning, the marginal value theorem and Bayesian inference approaches to offer a learning algorithm and a decision rule for making sequential stay-or-leave choices.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. A State Representation for Diminishing RewardsTed Moskovitz, Samo Hromadka, Ahmed Touati, Diana Borsa et al.NeurIPS 2023 · 4 citations

Related papers

  1. Dynamic allocation of limited memory resources in reinforcement learningNisheet Patel, Luigi Acerbi, Alexandre PougetNeurIPS 2020 · 6 citations
  2. Reinforcement Learning with Non-Exponential DiscountingMatthias Schultheis, Constantin A. Rothkopf, Heinz KoepplNeurIPS 2022 · 19 citations
  3. Flexible inference for animal learning rules using neural networksYuhan Helena Liu, Victor Geadah, Jonathan W. PillowNeurIPS 2025
  4. Inverse Rational Control with Partially Observable Continuous Nonlinear DynamicsMinhae Kwon, Saurabh Daptardar, Paul R. Schrater, Xaq PitkowNeurIPS 2020 · 46 citations
  5. Learning to Stop: Deep Learning for Mean Field Optimal StoppingLorenzo Magnino, Yuchen Zhu, Mathieu LaurièreICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines