  Out-of-Distribution (OOD) generalization, a cornerstone for building robust
machine learning models capable of handling data diverging from the training
set's distribution, is an ongoing challenge in deep learning. While significant
progress has been observed in computer vision and natural language processing,
its exploration in tabular data, ubiquitous in many industrial applications,
remains nascent. To bridge this gap, we present Wild-Tab, a large-scale
benchmark tailored for OOD generalization in tabular regression tasks. The
benchmark incorporates 3 industrial datasets sourced from fields like weather
prediction and power consumption estimation, providing a challenging testbed
for evaluating OOD performance under real-world conditions. Our extensive
experiments, evaluating 10 distinct OOD generalization methods on Wild-Tab,
reveal nuanced insights. We observe that many of these methods often struggle
to maintain high-performance levels on unseen data, with OOD performance
showing a marked drop compared to in-distribution performance. At the same
time, Empirical Risk Minimization (ERM), despite its simplicity, delivers
robust performance across all evaluations, rivaling the results of
state-of-the-art methods. Looking forward, we hope that the release of Wild-Tab
will facilitate further research on OOD generalization and aid in the
deployment of machine learning models in various real-world contexts where
handling distribution shifts is a crucial requirement.
