  Open-source benchmark datasets have been a critical component for advancing
machine learning for robot perception in terrestrial applications. Benchmark
datasets enable the widespread development of state-of-the-art machine learning
methods, which require large datasets for training, validation, and thorough
comparison to competing approaches. Underwater environments impose several
operational challenges that hinder efforts to collect large benchmark datasets
for marine robot perception. Furthermore, a low abundance of targets of
interest relative to the size of the search space leads to increased time and
cost required to collect useful datasets for a specific task. As a result,
there is limited availability of labeled benchmark datasets for underwater
applications. We present the AI4Shipwrecks dataset, which consists of 24
distinct shipwreck sites totaling 286 high-resolution labeled side scan sonar
images to advance the state-of-the-art in autonomous sonar image understanding.
We leverage the unique abundance of targets in Thunder Bay National Marine
Sanctuary in Lake Huron, MI, to collect and compile a sonar imagery benchmark
dataset through surveys with an autonomous underwater vehicle (AUV). We
consulted with expert marine archaeologists for the labeling of robotically
gathered data. We then leverage this dataset to perform benchmark experiments
for comparison of state-of-the-art supervised segmentation methods, and we
present insights on opportunities and open challenges for the field. The
dataset and benchmarking tools will be released as an open-source benchmark
dataset to spur innovation in machine learning for Great Lakes and ocean
exploration. The dataset and accompanying software are available at
https://umfieldrobotics.github.io/ai4shipwrecks/.
