A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions

Estimating value functions is a core component of reinforcement learning algorithms. Temporal difference (TD) learning algorithms use bootstrapping, i.e. they update the value function toward a learning target using value estimates at subsequent time-steps. Alternatively, the value function can be updated toward a learning target constructed by separately predicting successor features (SF)-a policy-dependent modeland linearly combining them with instantaneous rewards. We focus on bootstrapping targets used when estimating value functions, and propose a new backup target, the η-return mixture, which implicitly combines value-predictive knowledge (used by TD methods) with (successor) feature-predictive knowledge-with a parameter η capturing how much to rely on each. We illustrate that incorporating predictive knowledge through an ηγ-discounted SF model makes more efficient use of sampled experience, compared to either extreme, i.e. bootstrapping entirely on the value function estimate, or bootstrapping on the product of separately estimated successor features and instantaneous reward models. We empirically show this approach leads to faster policy evaluation and better control performance, for tabular and nonlinear function approximations, indicating scalability and generality. The fundamental goal of reinforcement learning (RL) is to maximize return, i.e. (temporally discounted) cumulative reward. Value functions provide an estimate of the expected return from a specific state (and action), and as such, they are a fundamental component of RL algorithms. Modern deep RL methods require numerous environment interactions to solve complex tasks, which can be expensive or impossible to obtain, particularly for tasks resembling the real-world. This makes it essential to develop data-efficient methods for learning accurate value functions. The problem we address in this work is that of credit assignment, namely how to associate (distant) rewards to the states and actions that caused them. Value-based RL methods tackle this problem through temporal difference (TD) learning algorithms (Sutton 1988). TD algorithms rely on bootstrapping: using the value estimate at a subsequent timestep, together with the observed data (e.g. rewards), to construct the learning target-the return-for the current timestep. However, the value estimate in the backup target does not need to come from the current value function