Lune

NeurIPS2021Top-tier venue

A Minimalist Approach to Offline Reinforcement Learning

Scott Fujimoto, Shixiang Shane Gu

Abstract

Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

438 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • COMBO: Conservative Offline Model-Based Policy Optimization cites Conservative Q-Learning for Offline Reinforcement Learning
  • COMBO: Conservative Offline Model-Based Policy Optimization cites Offline Reinforcement Learning with Fisher Divergence Critic Regularization
  • Contrastive Learning as Goal-Conditioned Reinforcement Learning cites Offline Reinforcement Learning with Implicit Q-Learning
  • Contrastive Learning as Goal-Conditioned Reinforcement Learning cites Decision Transformer: Reinforcement Learning via Sequence Modeling
  • Online Decision Transformer cites Offline Reinforcement Learning with Implicit Q-Learning
  • Online Decision Transformer cites Decision Transformer: Reinforcement Learning via Sequence Modeling
  • Online Decision Transformer cites Conservative Q-Learning for Offline Reinforcement Learning
  • Online Decision Transformer cites Offline Reinforcement Learning with Fisher Divergence Critic Regularization
  • Offline Reinforcement Learning with Implicit Q-Learning cites Decision Transformer: Reinforcement Learning via Sequence Modeling
  • Offline Reinforcement Learning with Implicit Q-Learning cites Conservative Q-Learning for Offline Reinforcement Learning
  • Offline Reinforcement Learning with Implicit Q-Learning cites Offline Reinforcement Learning with Fisher Divergence Critic Regularization
  • Decision Transformer: Reinforcement Learning via Sequence Modeling cites Conservative Q-Learning for Offline Reinforcement Learning
  • Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning cites Online Decision Transformer
  • Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning cites Offline Reinforcement Learning with Implicit Q-Learning
  • Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning cites Conservative Q-Learning for Offline Reinforcement Learning
  • Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning cites Offline Reinforcement Learning with Fisher Divergence Critic Regularization
  • Offline Reinforcement Learning with Fisher Divergence Critic Regularization cites Conservative Q-Learning for Offline Reinforcement Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers438

  1. Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey LevineICLR 2022 · 1,402 citations
  2. COMBO: Conservative Offline Model-Based Policy OptimizationTianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran et al.NeurIPS 2021 · 549 citations
  3. Contrastive Learning as Goal-Conditioned Reinforcement LearningBenjamin Eysenbach, Tianjun Zhang, Sergey Levine, Ruslan SalakhutdinovNeurIPS 2022 · 331 citations
  4. Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-TuningMitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark et al.NeurIPS 2023 · 296 citations
  5. Online Decision TransformerQinqing Zheng, Amy Zhang, Aditya GroverICML 2022 · 256 citations

Builds on14

  1. Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker, Sergey LevineNeurIPS 2020 · 2,881 citations
  2. Decision Transformer: Reinforcement Learning via Sequence ModelingLili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee et al.NeurIPS 2021 · 2,557 citations
  3. Critic Regularized RegressionZiyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel et al.NeurIPS 2020 · 406 citations
  4. Offline Reinforcement Learning with Fisher Divergence Critic RegularizationIlya Kostrikov, Rob Fergus, Jonathan Tompson, Ofir NachumICML 2021 · 350 citations
  5. Implementation Matters in Deep RL: A Case Study on PPO and TRPOLogan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras et al.ICLR 2020 · 305 citations

Related papers

  1. Behavior Proximal Policy OptimizationZifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang et al.ICLR 2023 · 8 citations
  2. Should I Run Offline Reinforcement Learning or Behavioral Cloning?Aviral Kumar, Joey Hong, Anikait Singh, Sergey LevineICLR 2022 · 84 citations
  3. Offline RL Without Off-Policy EvaluationDavid Brandfonbrener, Will Whitney, Rajesh Ranganath, Joan BrunaNeurIPS 2021 · 217 citations
  4. Is Value Learning Really the Main Bottleneck in Offline RL?Seohong Park, Kevin Frans, Sergey Levine, Aviral KumarNeurIPS 2024 · 99 citations
  5. Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement LearningShentao Yang, Yihao Feng, Shujian Zhang, Mingyuan ZhouICML 2022 · 14 citations
Dusk over the sea between two cliffs drawn in fine vertical lines