  Despite the remarkable performance that modern deep neural networks have
achieved on independent and identically distributed (I.I.D.) data, they can
crash under distribution shifts. Most current evaluation methods for domain
generalization (DG) adopt the leave-one-out strategy as a compromise on the
limited number of domains. We propose a large-scale benchmark with extensive
labeled domains named NICO++ along with more rational evaluation methods for
comprehensively evaluating DG algorithms. To evaluate DG datasets, we propose
two metrics to quantify covariate shift and concept shift, respectively. Two
novel generalization bounds from the perspective of data construction are
proposed to prove that limited concept shift and significant covariate shift
favor the evaluation capability for generalization. Through extensive
experiments, NICO++ shows its superior evaluation capability compared with
current DG datasets and its contribution in alleviating unfairness caused by
the leak of oracle knowledge in model selection.
