  We study the problem of policy optimization for infinite-horizon discounted
Markov Decision Processes with softmax policy and nonlinear function
approximation trained with policy gradient algorithms. We concentrate on the
training dynamics in the mean-field regime, modeling e.g., the behavior of wide
single hidden layer neural networks, when exploration is encouraged through
entropy regularization. The dynamics of these models is established as a
Wasserstein gradient flow of distributions in parameter space. We further prove
global optimality of the fixed points of this dynamics under mild conditions on
their initialization.
