  State-of-the-art natural language processing (NLP) models often learn to
model dataset biases and surface form correlations instead of features that
target the intended underlying task. Previous work has demonstrated effective
methods to circumvent these issues when knowledge of the bias is available. We
consider cases where the bias issues may not be explicitly identified, and show
a method for training models that learn to ignore these problematic
correlations. Our approach relies on the observation that models with limited
capacity primarily learn to exploit biases in the dataset. We can leverage the
errors of such limited capacity models to train a more robust model in a
product of experts, thus bypassing the need to hand-craft a biased model. We
show the effectiveness of this method to retain improvements in
out-of-distribution settings even if no particular bias is targeted by the
biased model.
