  In order for artificial agents to coordinate effectively with people, they
must act consistently with existing conventions (e.g. how to navigate in
traffic, which language to speak, or how to coordinate with teammates). A
group's conventions can be viewed as a choice of equilibrium in a coordination
game. We consider the problem of an agent learning a policy for a coordination
game in a simulated environment and then using this policy when it enters an
existing group. When there are multiple possible conventions we show that
learning a policy via multi-agent reinforcement learning (MARL) is likely to
find policies which achieve high payoffs at training time but fail to
coordinate with the real group into which the agent enters. We assume access to
a small number of samples of behavior from the true convention and show that we
can augment the MARL objective to help it find policies consistent with the
real group's convention. In three environments from the literature - traffic,
communication, and team coordination - we observe that augmenting MARL with a
small amount of imitation learning greatly increases the probability that the
strategy found by MARL fits well with the existing social convention. We show
that this works even in an environment where standard training methods very
rarely find the true convention of the agent's partners.
