  Much of recent progress in NLU was shown to be due to models' learning
dataset-specific heuristics. We conduct a case study of generalization in NLI
(from MNLI to the adversarially constructed HANS dataset) in a range of
BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as
well as with subsampling the data and increasing the model size. We report 2
successful and 3 unsuccessful strategies, all providing insights into how
Transformer-based models learn to generalize.
