  "Effective robustness" measures the extra out-of-distribution (OOD)
robustness beyond what can be predicted from the in-distribution (ID)
performance. Existing effective robustness evaluations typically use a single
test set such as ImageNet to evaluate the ID accuracy. This becomes problematic
when evaluating models trained on different data distributions, e.g., comparing
models trained on ImageNet vs. zero-shot language-image pre-trained models
trained on LAION. In this paper, we propose a new evaluation metric to evaluate
and compare the effective robustness of models trained on different data. To do
this, we control for the accuracy on multiple ID test sets that cover the
training distributions for all the evaluated models. Our new evaluation metric
provides a better estimate of effective robustness when there are models with
different training data. It may also explain the surprising effective
robustness gains of zero-shot CLIP-like models exhibited in prior works that
used ImageNet as the only ID test set, while the gains diminish under our new
evaluation. Additional artifacts including interactive visualizations are
provided at https://shizhouxing.github.io/effective-robustness.
