  This paper presents an algorithmic framework for learning robust policies in
asymmetric imperfect-information games, where the joint reward could depend on
the uncertain opponent type (a private information known only to the opponent
itself and its ally). In order to maximize the reward, the protagonist agent
has to infer the opponent type through agent modeling. We use multiagent
reinforcement learning (MARL) to learn opponent models through self-play, which
captures the full strategy interaction and reasoning between agents. However,
agent policies learned from self-play can suffer from mutual overfitting.
Ensemble training methods can be used to improve the robustness of agent policy
against different opponents, but it also significantly increases the
computational overhead. In order to achieve a good trade-off between the
robustness of the learned policy and the computation complexity, we propose to
train a separate opponent policy against the protagonist agent for evaluation
purposes. The reward achieved by this opponent is a noisy measure of the
robustness of the protagonist agent policy due to the intrinsic stochastic
nature of a reinforcement learner. To handle this stochasticity, we apply a
stochastic optimization scheme to dynamically update the opponent ensemble to
optimize an objective function that strikes a balance between robustness and
computation complexity. We empirically show that, under the same limited
computational budget, the proposed method results in more robust policy
learning than standard ensemble training.
