  Dataset bias has attracted increasing attention recently for its detrimental
effect on the generalization ability of fine-tuned models. The current
mainstream solution is designing an additional shallow model to pre-identify
biased instances. However, such two-stage methods scale up the computational
complexity of training process and obstruct valid feature information while
mitigating bias. To address this issue, we utilize the representation
normalization method which aims at disentangling the correlations between
features of encoded sentences. We find it also promising in eliminating the
bias problem by providing isotropic data distribution. We further propose
Kernel-Whitening, a Nystrom kernel approximation method to achieve more
thorough debiasing on nonlinear spurious correlations. Our framework is
end-to-end with similar time consumption to fine-tuning. Experiments show that
Kernel-Whitening significantly improves the performance of BERT on
out-of-distribution datasets while maintaining in-distribution accuracy.
