Lune

CVPR2024Top-tier venue

Regularized Parameter Uncertainty for Improving Generalization in Reinforcement Learning

Pehuen Moure, Longbiao Cheng, Joachim Ott, Zuowen Wang, Shih-Chii Liu

Abstract

In order for reinforcement learning (RL) agents to be deployed in real-world environments, they must be able to generalize to unseen environments. However, RL struggles with out-of-distribution generalization, often due to overfitting the particulars of the training environment. Although regularization techniques from supervised learning can be applied to avoid over-fitting, the differences between supervised learning and RL limit their application. To address this, we propose the Signal-to-Noise Ratio regulated Parameter Uncertainty Network (SNR PUN) for RL. We introduce SNR as a new measure of regularizing the parameter uncertainty of a network and provide a formal analysis explaining why SNR regularization works well for RL. We demonstrate the effectiveness of our proposed method to generalize in several simulated environments; and in a physical system showing the possibility of using SNR PUN for applying RL to real-world applications.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on3

  1. Maximum Entropy RL (Provably) Solves Some Robust RL ProblemsBenjamin Eysenbach, Sergey LevineICLR 2022 · 244 citations
  2. Generalized Variational Continual LearningNoel Loo, Siddharth Swaroop, Richard E. TurnerICLR 2021 · 74 citations
  3. On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm PerspectiveZeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato et al.NeurIPS 2023 · 38 citations

Related papers

  1. Network Randomization: A Simple Technique for Generalization in Deep Reinforcement LearningKimin Lee, Kibok Lee, Jinwoo Shin, Honglak LeeICLR 2020 · 191 citations
  2. Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few DemonstrationsYujie Zhu, Charles A. Hepburn, Matthew Thorpe, Giovanni MontanaNeurIPS 2025
  3. Improving Generalization in Reinforcement Learning with Mixture RegularizationKaixin Wang, Bingyi Kang, Jie Shao, Jiashi FengNeurIPS 2020 · 143 citations
  4. Domain Generalization Guided by Gradient Signal to Noise Ratio of ParametersMateusz Michalkiewicz, Masoud Faraki, Xiang Yu, Manmohan Chandraker et al.ICCV 2023 · 9 citations
  5. Learning Domain Invariant Representations in Goal-conditioned Block MDPsBeining Han, Chongyi Zheng, Harris Chan, Keiran Paster et al.NeurIPS 2021 · 20 citations
Dusk over the sea between two cliffs drawn in fine vertical lines