Adaptive Pairwise Weights for Temporal Credit Assignment
Zeyu Zheng, Risto Vuorio, Richard L. Lewis, Satinder Singh
Abstract
How much credit (or blame) should an action taken in a state get for a future reward? This is the fundamental temporal credit assignment problem in Reinforcement Learning (RL). One of the earliest and still most widely used heuristics is to assign this credit based on a scalar coefficient, lambda (treated as a hyperparameter), raised to the power of the time interval between the state-action and the reward. In this empirical paper, we explore heuristics based on more general pairwise weightings that are functions of the state in which the action was taken, the state at the time of the reward, as well as the time interval between the two. Of course it isn't clear what these pairwise weight functions should be, and because they are too complex to be treated as hyperparameters we develop a metagradient procedure for learning these weight functions during the usual RL training of a policy. Our empirical work shows that it is often possible to learn these pairwise weight functions during learning of the policy to achieve better performance than competing approaches.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Discovering Reinforcement Learning Algorithms cites A Self-Tuning Actor-Critic Algorithm
- Discovering Reinforcement Learning Algorithms cites Meta-Gradient Reinforcement Learning with an Objective Discovered Online
- Meta-Gradient Reinforcement Learning with an Objective Discovered Online cites Discovering Reinforcement Learning Algorithms
- Discovering Reinforcement Learning Algorithms cites Behaviour Suite for Reinforcement Learning
- Discovering Reinforcement Learning Algorithms cites What Can Learned Intrinsic Rewards Capture?
Cited by top-tier papers1
Builds on7
- Behaviour Suite for Reinforcement LearningIan Osband, Yotam Doron, Matteo Hessel, John Aslanides et al.ICLR 2020 · 204 citations
- Discovering Reinforcement Learning AlgorithmsJunhyuk Oh, Matteo Hessel, Wojciech M. Czarnecki, Zhongwen Xu et al.NeurIPS 2020 · 154 citations
- A Self-Tuning Actor-Critic AlgorithmTom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel et al.NeurIPS 2020 · 106 citations
- Meta-Gradient Reinforcement Learning with an Objective Discovered OnlineZhongwen Xu, Hado Philip van Hasselt, Matteo Hessel, Junhyuk Oh et al.NeurIPS 2020 · 90 citations
- What Can Learned Intrinsic Rewards Capture?Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu et al.ICML 2020 · 87 citations
Related papers
- Learning Task-Distribution Reward Shaping with Meta-LearningHaosheng Zou, Tongzheng Ren, Dong Yan, Hang Su et al.AAAI 2021 · 19 citations
- Counterfactual Credit Assignment in Model-Free Reinforcement LearningThomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor et al.ICML 2021 · 70 citations
- Learning Guidance Rewards with Trajectory-space SmoothingTanmay Gangwani, Yuan Zhou, Jian PengNeurIPS 2020 · 46 citations
- Quantile Credit AssignmentThomas Mesnard, Wenqi Chen, Alaa Saade, Yunhao Tang et al.ICML 2023 · 3 citations
- Difference Advantage Estimation for Multi-Agent Policy GradientsYueheng Li, Guangming Xie, Zongqing LuICML 2022 · 24 citations
