  Despite the notable successes in video games such as Atari 2600, current AI
is yet to defeat human champions in the domain of real-time strategy (RTS)
games. One of the reasons is that an RTS game is a multi-agent game, in which
single-agent reinforcement learning methods cannot simply be applied because
the environment is not a stationary Markov Decision Process. In this paper, we
present a first step toward finding a game-theoretic solution to RTS games by
applying Neural Fictitious Self-Play (NFSP), a game-theoretic approach for
finding Nash equilibria, to Mini-RTS, a small but nontrivial RTS game provided
on the ELF platform. More specifically, we show that NFSP can be effectively
combined with policy gradient reinforcement learning and be applied to
Mini-RTS. Experimental results also show that the scalability of NFSP can be
substantially improved by pretraining the models with simple self-play using
policy gradients, which by itself gives a strong strategy despite its lack of
theoretical guarantee of convergence.
