Emphatic Algorithms for Deep Reinforcement Learning

Off-policy learning allows us to learn about possible policies of behavior from experience generated by a different behavior policy. Temporal difference (TD) learning algorithms can become unstable when combined with function approximation and off-policy sampling-this is known as the "deadly triad". Emphatic temporal difference (ETD(λ)) algorithm ensures convergence in the linear case by appropriately weighting the TD(λ) updates. In this paper, we extend the use of emphatic methods to deep reinforcement learning agents. We show that naively adapting ETD(λ) to popular deep reinforcement learning algorithms, which use forward view multi-step returns, results in poor performance. We then derive new emphatic algorithms for use in the context of such algorithms, and we demonstrate that they provide noticeable benefits in small problems designed to highlight the instability of TD methods. Finally, we observed improved performance when applying these algorithms at scale on classic Atari games from the Arcade Learning Environment. Off-policy learning, whereby an agent learns from behavior that differs from its current policy, affords an agent opportunities to accumulate rich knowledge (Degris & Modayil, 2012 ) by learning about the effect of different policies of behaviors. This can also be extended to learn about different goals, e.g., by learning general value functions (Sutton et al., 2011) for cumulants that differ from the main task reward. Unfortunately, it is well known that reinforcement learning algorithms (Sutton & Barto, 2018) can become unstable when combining function approximation, off-policy learning, and bootstrapping (Tsitsiklis & Van Roy, 1997)-for this reason such combination is referred to as the deadly triad (Sutton & Barto, 2018; van Hasselt et al., 2018) .