  Distribution shift presents a significant challenge in machine learning,
where models often underperform during the test stage when faced with a
different distribution than the one they were trained on. This paper focuses on
domain shifts, which occur when the model is applied to new domains that are
different from the ones it was trained on, and propose a new approach called
D$^3$G. Unlike previous methods that aim to learn a single model that is domain
invariant, D$^3$G leverages domain similarities based on domain metadata to
learn domain-specific models. Concretely, D$^3$G learns a set of
training-domain-specific functions during the training stage and reweights them
based on domain relations during the test stage. These domain relations can be
directly obtained and learned from domain metadata. Under mild assumptions, we
theoretically prove that using domain relations to reweight
training-domain-specific functions achieves stronger out-of-domain
generalization compared to the conventional averaging approach. Empirically, we
evaluate the effectiveness of D$^3$G using real-world datasets for tasks such
as temperature regression, land use classification, and molecule-protein
binding affinity prediction. Our results show that D$^3$G consistently
outperforms state-of-the-art methods.
