  Neural network models have been very successful in natural language
inference, with the best models reaching 90% accuracy in some benchmarks.
However, the success of these models turns out to be largely benchmark
specific. We show that models trained on a natural language inference dataset
drawn from one benchmark fail to perform well in others, even if the notion of
inference assumed in these benchmarks is the same or similar. We train six high
performing neural network models on different datasets and show that each one
of these has problems of generalizing when we replace the original test set
with a test set taken from another corpus designed for the same task. In light
of these results, we argue that most of the current neural network models are
not able to generalize well in the task of natural language inference. We find
that using large pre-trained language models helps with transfer learning when
the datasets are similar enough. Our results also highlight that the current
NLI datasets do not cover the different nuances of inference extensively
enough.
