  Multi agent strategies in mixed cooperative-competitive environments can be
hard to craft by hand because each agent needs to coordinate with its teammates
while competing with its opponents. Learning based algorithms are appealing but
many scenarios require heterogeneous agent behavior for the team's success and
this increases the complexity of the learning algorithm. In this work, we
develop a competitive multi agent environment called FortAttack in which two
teams compete against each other. We corroborate that modeling agents with
Graph Neural Networks and training them with Reinforcement Learning leads to
the evolution of increasingly complex strategies for each team. We observe a
natural emergence of heterogeneous behavior amongst homogeneous agents when
such behavior can lead to the team's success. Such heterogeneous behavior from
homogeneous agents is appealing because any agent can replace the role of
another agent at test time. Finally, we propose ensemble training, in which we
utilize the evolved opponent strategies to train a single policy for friendly
agents.
