  Fine-tuning pretrained models is a common practice in domain generalization
(DG) tasks. However, fine-tuning is usually computationally expensive due to
the ever-growing size of pretrained models. More importantly, it may cause
over-fitting on source domain and compromise their generalization ability as
shown in recent works. Generally, pretrained models possess some level of
generalization ability and can achieve decent performance regarding specific
domains and samples. However, the generalization performance of pretrained
models could vary significantly over different test domains even samples, which
raises challenges for us to best leverage pretrained models in DG tasks. In
this paper, we propose a novel domain generalization paradigm to better
leverage various pretrained models, named specialized ensemble learning for
domain generalization (SEDGE). It first trains a linear label space adapter
upon fixed pretrained models, which transforms the outputs of the pretrained
model to the label space of the target domain. Then, an ensemble network aware
of model specialty is proposed to dynamically dispatch proper pretrained models
to predict each test sample. Experimental studies on several benchmarks show
that SEDGE achieves significant performance improvements comparing to strong
baselines including state-of-the-art method in DG tasks and reduces the
trainable parameters by ~99% and the training time by ~99.5%.
