Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime

We study the problem of policy optimization for infinite-horizon discounted Markov Decision with softmax policy and nonlinear function approximation trained with policy algorithms. We concentrate on the training dynamics in the mean-field regime, e.g., the behavior of wide single hidden layer neural networks, when exploration encouraged through entropy regularization. The dynamics of these models is established a Wasserstein gradient flow of distributions in parameter space. We further prove global of the fixed points of this dynamics under mild conditions on their initialization.