  An interesting iterative procedure is proposed to solve a two-player zero-sum
Markov games. First this problem is expressed as a min-max Markov game. Next, a
two-step Q-learning algorithm for solving Markov decision problem (MDP) is
suitably modified to solve this Markov game. Under a suitable assumption, the
boundedness of the proposed iterates is obtained theoretically. Using results
from stochastic approximation, the almost sure convergence of the proposed
two-step minimax Q-learning is obtained theoretically. More specifically, the
proposed algorithm converges to the game theoretic optimal value with
probability one, when the model information is not known. Numerical simulation
authenticate that the proposed algorithm is effective and easy to implement.
