Value Improved Actor Critic Algorithms

To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms. On the other hand, slow changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability. To better address this tradeoff, we propose to decouple the acting policy from the policy evaluated by the critic. This allows the agent to separately improve the critic's policy (e.g. value improvement) with greedier updates while maintaining the slow gradient-based improvement to the parameterized acting policy. We investigate the convergence of this approach in the finite-horizon domain using a popular analysis scheme which generalizes Policy Iteration with arbitrary improvement operators and approximate evaluation. Empirically, incorporating value-improvement into the popular off-policy actorcritic algorithms TD3 and SAC significantly improves or matches performance over the baselines respectively, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost 1 .