  Many natural language inference (NLI) datasets contain biases that allow
models to perform well by only using a biased subset of the input, without
considering the remainder features. For instance, models are able to make a
classification decision by only using the hypothesis, without learning the true
relationship between it and the premise. These structural biases lead
discriminative models to learn unintended superficial features and to
generalize poorly out of the training distribution. In this work, we
reformulate the NLI task as a generative task, where a model is conditioned on
the biased subset of the input and the label and generates the remaining subset
of the input. We show that by imposing a uniform prior, we obtain a provably
unbiased model. Through synthetic experiments, we find that this approach is
highly robust to large amounts of bias. We then demonstrate empirically on two
types of natural bias that this approach leads to fully unbiased models in
practice. However, we find that generative models are difficult to train and
they generally perform worse than discriminative baselines. We highlight the
difficulty of the generative modeling task in the context of NLI as a cause for
this worse performance. Finally, by fine-tuning the generative model with a
discriminative objective, we reduce the performance gap between the generative
model and the discriminative baseline, while allowing for a small amount of
bias.
