  Changes to hyperparameters can have a dramatic effect on model accuracy.
Thus, the tuning of hyperparameters plays an important role in optimizing
machine-learning models. An integral part of the hyperparameter-tuning process
is the evaluation of model checkpoints, which is done through the use of
"validators". In a supervised setting, these validators evaluate checkpoints by
computing accuracy on a validation set that has labels. In contrast, in an
unsupervised setting, the validation set has no such labels. Without any
labels, it is impossible to compute accuracy, so validators must estimate
accuracy instead. But what is the best approach to estimating accuracy? In this
paper, we consider this question in the context of unsupervised domain
adaptation (UDA). Specifically, we propose three new validators, and we compare
and rank them against five other existing validators, on a large dataset of
1,000,000 checkpoints. Extensive experimental results show that two of our
proposed validators achieve state-of-the-art performance in various settings.
Finally, we find that in many cases, the state-of-the-art is obtained by a
simple baseline method. To the best of our knowledge, this is the largest
empirical study of UDA validators to date. Code is available at
https://www.github.com/KevinMusgrave/powerful-benchmarker.
