  We propose a multi-agent reinforcement learning dynamics, and analyze its
convergence in infinite-horizon discounted Markov potential games. We focus on
the independent and decentralized setting, where players do not have knowledge
of the game model and cannot coordinate. In each stage, players update their
estimate of Q-function that evaluates their total contingent payoff based on
the realized one-stage reward in an asynchronous manner. Then, players
independently update their policies by incorporating an optimal one-stage
deviation strategy based on the estimated Q-function. A key feature of the
learning dynamics is that the Q-function estimates are updated at a faster
timescale than the policies. We prove that the policies induced by our learning
dynamics converge to the set of stationary Nash equilibria in Markov potential
games with probability 1. Our results highlight the efficacy of simple learning
dynamics in reaching to the set of stationary Nash equilibrium even in
environments with minimal information available.
