  The objective of this work is to evaluate multi-agent artificial intelligence
methods when deployed on teams of unmanned surface vehicles (USV) in an
adversarial environment. Autonomous agents were evaluated in real-world
scenarios using the Aquaticus test-bed, which is a Capture-the-Flag (CTF) style
competition involving teams of USV systems. Cooperative teaming algorithms of
various foundations in behavior-based optimization and deep reinforcement
learning (RL) were deployed on these USV systems in two versus two teams and
tested against each other during a competition period in the fall of 2023. Deep
reinforcement learning applied to USV agents was achieved via the Pyquaticus
test bed, a lightweight gymnasium environment that allows simulated CTF
training in a low-level environment. The results of the experiment demonstrate
that rule-based cooperation for behavior-based agents outperformed those
trained in Deep-reinforcement learning paradigms as implemented in these
competitions. Further integration of the Pyquaticus gymnasium environment for
RL with MOOS-IvP in terms of configuration and control schema will allow for
more competitive CTF games in future studies. As the development of
experimental deep RL methods continues, the authors expect that the competitive
gap between behavior-based autonomy and deep RL will be reduced. As such, this
report outlines the overall competition, methods, and results with an emphasis
on future works such as reward shaping and sim-to-real methodologies and
extending rule-based cooperation among agents to react to safety and security
events in accordance with human experts intent/rules for executing safety and
security processes.
