  Softmax policy gradient is a popular algorithm for policy optimization in
single-agent reinforcement learning, particularly since projection is not
needed for each gradient update. However, in multi-agent systems, the lack of
central coordination introduces significant additional difficulties in the
convergence analysis. Even for a stochastic game with identical interest, there
can be multiple Nash Equilibria (NEs), which disables proof techniques that
rely on the existence of a unique global optimum. Moreover, the softmax
parameterization introduces non-NE policies with zero gradient, making it
difficult for gradient-based algorithms in seeking NEs. In this paper, we study
the finite time convergence of decentralized softmax gradient play in a special
form of game, Markov Potential Games (MPGs), which includes the identical
interest game as a special case. We investigate both gradient play and natural
gradient play, with and without $\log$-barrier regularization. The established
convergence rates for the unregularized cases contain a trajectory-dependent
constant that can be arbitrarily large, whereas the $\log$-barrier
regularization overcomes this drawback, with the cost of slightly worse
dependence on other factors such as the action set size. An empirical study on
an identical interest matrix game confirms the theoretical findings.
