  Crater mapping using neural networks and other automated methods has
increased recently with automated Crater Detection Algorithms (CDAs) applied to
planetary bodies throughout the solar system. A recent publication by Benedix
et al. (2020) showed high performance at small scales compared to similar
automated CDAs but with a net positive diameter bias in many crater candidates.
I compare the publicly available catalogs from Benedix et al. (2020) and Lee &
Hogan (2021) and show that the reported performance is sensitive to the metrics
used to test the catalogs. I show how the more permissive comparison methods
indicate a higher CDA performance by allowing worse candidate craters to match
ground-truth craters. I show that the Benedix et al. (2020) catalog has a
substantial performance loss with increasing latitude and identify an image
projection issue that might cause this loss. Finally, I suggest future
applications of neural networks in generating large scientific datasets be
validated using secondary networks with independent data sources or training
methods.
