  There are many AI tasks involving multiple interacting agents where agents
should learn to cooperate and collaborate to effectively perform the task. Here
we develop and evaluate various multi-agent protocols to train agents to
collaborate with teammates in grid soccer. We train and evaluate our
multi-agent methods against a team operating with a smart hand-coded policy. As
a baseline, we train agents concurrently and independently, with no
communication. Our collaborative protocols were parameter sharing, coordinated
learning with communication, and counterfactual policy gradients. Against the
hand-coded team, the team trained with parameter sharing and the team trained
with coordinated learning performed the best, scoring on 89.5% and 94.5% of
episodes respectively when playing against the hand-coded team. Against the
parameter sharing team, with adversarial training the coordinated learning team
scored on 75% of the episodes, indicating it is the most adaptable of our
methods. The insights gained from our work can be applied to other domains
where multi-agent collaboration could be beneficial.
