  Designing robust machine learning systems remains an open problem, and there
is a need for benchmark problems that cover both environmental changes and
evaluation on a downstream task. In this work, we introduce AVOIDDS, a
realistic object detection benchmark for the vision-based aircraft
detect-and-avoid problem. We provide a labeled dataset consisting of 72,000
photorealistic images of intruder aircraft with various lighting conditions,
weather conditions, relative geometries, and geographic locations. We also
provide an interface that evaluates trained models on slices of this dataset to
identify changes in performance with respect to changing environmental
conditions. Finally, we implement a fully-integrated, closed-loop simulator of
the vision-based detect-and-avoid problem to evaluate trained models with
respect to the downstream collision avoidance task. This benchmark will enable
further research in the design of robust machine learning systems for use in
safety-critical applications. The AVOIDDS dataset and code are publicly
available at https://purl.stanford.edu/hj293cv5980 and
https://github.com/sisl/VisionBasedAircraftDAA respectively.
