  The release of large natural language inference (NLI) datasets like SNLI and
MNLI have led to rapid development and improvement of completely neural systems
for the task. Most recently, heavily pre-trained, Transformer-based models like
BERT and MT-DNN have reached near-human performance on these datasets. However,
these standard datasets have been shown to contain many annotation artifacts,
allowing models to shortcut understanding using simple fallible heuristics, and
still perform well on the test set. So it is no surprise that many adversarial
(challenge) datasets have been created that cause models trained on standard
datasets to fail dramatically. Although extra training on this data generally
improves model performance on just that type of data, transferring that
learning to unseen examples is still partial at best. This work evaluates the
failures of state-of-the-art models on existing adversarial datasets that test
different linguistic phenomena, and find that even though the models perform
similarly on MNLI, they differ greatly in their robustness to these attacks. In
particular, we find syntax-related attacks to be particularly effective across
all models, so we provide a fine-grained analysis and comparison of model
performance on those examples. We draw conclusions about the value of model
size and multi-task learning (beyond comparing their standard test set
performance), and provide suggestions for more effective training data.
