  When neural networks are trained on biased datasets, they tend to
inadvertently learn spurious correlations, leading to challenges in achieving
strong generalization and robustness. Current approaches to address such biases
typically involve utilizing bias annotations, reweighting based on pseudo-bias
labels, or enhancing diversity within bias-conflicting data points through
augmentation techniques. We introduce DeNetDM, a novel debiasing method based
on the observation that shallow neural networks prioritize learning core
attributes, while deeper ones emphasize biases when tasked with acquiring
distinct information. Using a training paradigm derived from Product of
Experts, we create both biased and debiased branches with deep and shallow
architectures and then distill knowledge to produce the target debiased model.
Extensive experiments and analyses demonstrate that our approach outperforms
current debiasing techniques, achieving a notable improvement of around 5% in
three datasets, encompassing both synthetic and real-world data. Remarkably,
DeNetDM accomplishes this without requiring annotations pertaining to bias
labels or bias types, while still delivering performance on par with supervised
counterparts. Furthermore, our approach effectively harnesses the diversity of
bias-conflicting points within the data, surpassing previous methods and
obviating the need for explicit augmentation-based methods to enhance the
diversity of such bias-conflicting points. The source code will be available
upon acceptance.
