  We explore the use of policy approximations to reduce the computational cost
of learning Nash equilibria in zero-sum stochastic games. We propose a new
Q-learning type algorithm that uses a sequence of entropy-regularized soft
policies to approximate the Nash policy during the Q-function updates. We prove
that under certain conditions, by updating the regularized Q-function, the
algorithm converges to a Nash equilibrium. We also demonstrate the proposed
algorithm's ability to transfer previous training experiences, enabling the
agents to adapt quickly to new environments. We provide a dynamic
hyper-parameter scheduling scheme to further expedite convergence. Empirical
results applied to a number of stochastic games verify that the proposed
algorithm converges to the Nash equilibrium, while exhibiting a major speed-up
over existing algorithms.
