  Large pre-trained language models have shown remarkable performance over the
past few years. These models, however, sometimes learn superficial features
from the dataset and cannot generalize to the distributions that are dissimilar
to the training scenario. There have been several approaches proposed to reduce
model's reliance on these bias features which can improve model robustness in
the out-of-distribution setting. However, existing methods usually use a fixed
low-capacity model to deal with various bias features, which ignore the
learnability of those features. In this paper, we analyze a set of existing
bias features and demonstrate there is no single model that works best for all
the cases. We further show that by choosing an appropriate bias model, we can
obtain a better robustness result than baselines with a more sophisticated
model design.
