  The objective of Domain Generalization (DG) is to devise algorithms and
models capable of achieving high performance on previously unseen test
distributions. In the pursuit of this objective, average measure has been
employed as the prevalent measure for evaluating models and comparing
algorithms in the existing DG studies. Despite its significance, a
comprehensive exploration of the average measure has been lacking and its
suitability in approximating the true domain generalization performance has
been questionable. In this study, we carefully investigate the limitations
inherent in the average measure and propose worst+gap measure as a robust
alternative. We establish theoretical grounds of the proposed measure by
deriving two theorems starting from two different assumptions. We conduct
extensive experimental investigations to compare the proposed worst+gap measure
with the conventional average measure. Given the indispensable need to access
the true DG performance for studying measures, we modify five existing datasets
to come up with SR-CMNIST, C-Cats&Dogs, L-CIFAR10, PACS-corrupted, and
VLCS-corrupted datasets. The experiment results unveil an inferior performance
of the average measure in approximating the true DG performance and confirm the
robustness of the theoretically supported worst+gap measure.
