A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning

Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of O 𝐾𝛼 𝐾 σIn |𝜏| -0.5 w.r.t. inner-loop update step 𝐾, learning rate 𝛼, estimate variance σ2 In and sample size |𝜏|, and 2) the multi-step Hessian estimation bias Δ𝐻 due to the use of autodiff, which has a polynomial impact O (𝐾 -1) ( Δ𝐻 ) 𝐾 -1 on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general. * Equal contribution, the order is determined by flipping a coin. See Appendix J for more details. † Corresponding author. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).