  We study the performance of the gradient play algorithm for stochastic games
(SGs), where each agent tries to maximize its own total discounted reward by
making decisions independently based on current state information which is
shared between agents. Policies are directly parameterized by the probability
of choosing a certain action at a given state. We show that Nash equilibria
(NEs) and first-order stationary policies are equivalent in this setting, and
give a local convergence rate around strict NEs. Further, for a subclass of SGs
called Markov potential games (which includes the setting with identical
rewards as an important special case), we design a sample-based reinforcement
learning algorithm and give a non-asymptotic global convergence rate analysis
for both exact gradient play and our sample-based learning algorithm. Our
result shows that the number of iterations to reach an $\epsilon$-NE scales
linearly, instead of exponentially, with the number of agents. Local geometry
and local stability are also considered, where we prove that strict NEs are
local maxima of the total potential function and fully-mixed NEs are saddle
points.
