On the Performance of Temporal Difference Learning With Neural Networks

Neural Temporal Difference (TD) Learning is an approximate temporal difference method for policy evaluation that uses a neural network for function approximation. Analysis of Neural TD Learning has proven to be challenging. In this paper we provide a convergence analysis of Neural TD Learning with a projection onto B(θ 0 , ω), a ball of fixed radius ω around the initial point θ 0 . We show an approximation bound of O(ϵ) + Õ(1/ √ m) where ϵ is the approximation quality of the best neural network in B(θ 0 , ω) and m is the width of all hidden layers in the network.