  Recent extensions to dynamic games of the well-known fictitious play learning
procedure in static games were proved to globally converge to stationary Nash
equilibria in two important classes of dynamic games (zero-sum and
identical-interest discounted stochastic games). However, those decentralized
algorithms need the players to know exactly the model (the transition
probabilities and their payoffs at every stage). To overcome these strong
assumptions, our paper introduces regularizations of the systems in (Leslie
2020; Baudin 2022) to construct a family of new decentralized learning
algorithms which are model-free (players don't know the transitions and their
payoffs are perturbed at every stage). Our procedures can be seen as extensions
to stochastic games of the classical smooth fictitious play learning procedures
in static games (where the players best responses are regularized, thanks to a
smooth strictly concave perturbation of their payoff functions). We prove the
convergence of our family of procedures to stationary regularized Nash
equilibria in zero-sum and identical-interest discounted stochastic games. The
proof uses the continuous smooth best-response dynamics counterparts, and
stochastic approximation methods. When there is only one player, our problem is
an instance of Reinforcement Learning and our procedures are proved to globally
converge to the optimal stationary policy of the regularized MDP. In that
sense, they can be seen as an alternative to the well known Q-learning
procedure.
