  Certain but important classes of strategic-form games, including zero-sum and
identical-interest games, have the fictitious-play-property (FPP), i.e.,
beliefs formed in fictitious play dynamics always converge to a Nash
equilibrium (NE) in the repeated play of these games. Such convergence results
are seen as a (behavioral) justification for the game-theoretical equilibrium
analysis. Markov games (MGs), also known as stochastic games, generalize the
repeated play of strategic-form games to dynamic multi-state settings with
Markovian state transitions. In particular, MGs are standard models for
multi-agent reinforcement learning -- a reviving research area in learning and
games, and their game-theoretical equilibrium analyses have also been conducted
extensively. However, whether certain classes of MGs have the FPP or not (i.e.,
whether there is a behavioral justification for equilibrium analysis or not)
remains largely elusive. In this paper, we study a new variant of fictitious
play dynamics for MGs and show its convergence to an NE in n-player
identical-interest MGs in which a single player controls the state transitions.
Such games are of interest in communications, control, and economics
applications. Our result together with the recent results in [Sayin et al.
2020] establishes the FPP of two-player zero-sum MGs and n-player
identical-interest MGs with a single controller (standing at two different ends
of the MG spectrum from fully competitive to fully cooperative).
