  Graph data are inherently complex and heterogeneous, leading to a high
natural diversity of distributional shifts. However, it remains unclear how to
build machine learning architectures that generalize to complex non-synthetic
distributional shifts naturally occurring in the real world. Here we develop
GraphMETRO, a Graph Neural Network architecture, that reliably models natural
diversity and captures complex distributional shifts. GraphMETRO employs a
Mixture-of-Experts (MoE) architecture with a gating model and multiple expert
models, where each expert model targets a specific distributional shift to
produce a shift-invariant representation, and the gating model identifies shift
components. Additionally, we design a novel objective that aligns the
representations from different expert models to ensure smooth optimization.
GraphMETRO achieves state-of-the-art results on four datasets from GOOD
benchmark comprised of complex and natural real-world distribution shifts,
improving by 67% and 4.2% on WebKB and Twitch datasets.
