  Imitation learning algorithms can be used to learn a policy from expert
demonstrations without access to a reward signal. However, most existing
approaches are not applicable in multi-agent settings due to the existence of
multiple (Nash) equilibria and non-stationary environments. We propose a new
framework for multi-agent imitation learning for general Markov games, where we
build upon a generalized notion of inverse reinforcement learning. We further
introduce a practical multi-agent actor-critic algorithm with good empirical
performance. Our method can be used to imitate complex behaviors in
high-dimensional environments with multiple cooperative or competing agents.
