  The choice of hyperparameters greatly impacts performance in natural language
processing. Often, it is hard to tell if a method is better than another or
just better tuned. Tuning curves fix this ambiguity by accounting for tuning
effort. Specifically, they plot validation performance as a function of the
number of hyperparameter choices tried so far. While several estimators exist
for these curves, it is common to use point estimates, which we show fail
silently and give contradictory results when given too little data.
  Beyond point estimates, confidence bands are necessary to rigorously
establish the relationship between different approaches. We present the first
method to construct valid confidence bands for tuning curves. The bands are
exact, simultaneous, and distribution-free, thus they provide a robust basis
for comparing methods.
  Empirical analysis shows that while bootstrap confidence bands, which serve
as a baseline, fail to approximate their target confidence, ours achieve it
exactly. We validate our design with ablations, analyze the effect of sample
size, and provide guidance on comparing models with our method. To promote
confident comparisons in future work, we release opda: an easy-to-use library
that you can install with pip. https://github.com/nicholaslourie/opda
