  In multi-agent reinforcement learning, the inherent non-stationarity of the
environment caused by other agents' actions posed significant difficulties for
an agent to learn a good policy independently. One way to deal with
non-stationarity is opponent modeling, by which the agent takes into
consideration the influence of other agents' policies. Most existing work
relies on predicting other agents' actions or goals, or discriminating between
different policies. However, such modeling fails to capture the similarities
and differences between policies simultaneously and thus cannot provide enough
useful information when generalizing to unseen agents. To address this, we
propose a general method to learn representations of other agents' policies,
such that the distance between policies is deliberately reflected by the
distance between representations, while the policy distance is inferred from
the sampled joint action distributions during training. We empirically show
that the agent conditioned on the learned policy representation can well
generalize to unseen agents in three multi-agent tasks.
