  Spurious correlations in training data significantly hinder the
generalization capability of machine learning models when faced with
distribution shifts in real-world scenarios. To tackle the problem, numerous
debias approaches have been proposed and benchmarked on datasets intentionally
designed with severe biases. However, it remains to be asked: \textit{1. Do
existing benchmarks really capture biases in the real world? 2. Can existing
debias methods handle biases in the real world?} To answer the questions, we
revisit biased distributions in existing benchmarks and real-world datasets,
and propose a fine-grained framework for analyzing dataset bias by
disentangling it into the magnitude and prevalence of bias. We observe and
theoretically demonstrate that existing benchmarks poorly represent real-world
biases. We further introduce two novel biased distributions to bridge this gap,
forming a nuanced evaluation framework for real-world debiasing. Building upon
these results, we evaluate existing debias methods with our evaluation
framework. Results show that existing methods are incapable of handling
real-world biases. Through in-depth analysis, we propose a simple yet effective
approach that can be easily applied to existing debias methods, named Debias in
Destruction (DiD). Empirical results demonstrate the superiority of DiD,
improving the performance of existing methods on all types of biases within the
proposed evaluation framework.
