  Machine learning methods are commonly evaluated and compared by their
performance on data sets from public repositories. This allows for multiple
methods, oftentimes several thousands, to be evaluated under identical
conditions and across time. The highest ranked performance on a problem is
referred to as state-of-the-art (SOTA) performance, and is used, among other
things, as a reference point for publication of new methods. Using the
highest-ranked performance as an estimate for SOTA is a biased estimator,
giving overly optimistic results. The mechanisms at play are those of
multiplicity, a topic that is well-studied in the context of multiple
comparisons and multiple testing, but has, as far as the authors are aware of,
been nearly absent from the discussion regarding SOTA estimates. The optimistic
state-of-the-art estimate is used as a standard for evaluating new methods, and
methods with substantial inferior results are easily overlooked. In this
article, we provide a probability distribution for the case of multiple
classifiers so that known analyses methods can be engaged and a better SOTA
estimate can be provided. We demonstrate the impact of multiplicity through a
simulated example with independent classifiers. We show how classifier
dependency impacts the variance, but also that the impact is limited when the
accuracy is high. Finally, we discuss three real-world examples; Kaggle
competitions that demonstrate various aspects.
