  We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently
introduced large-scale human-and-model-in-the-loop natural language inference
dataset collected over multiple rounds. We propose a fine-grained annotation
scheme of the different aspects of inference that are responsible for the gold
classification labels, and use it to hand-code all three of the ANLI
development sets. We use these annotations to answer a variety of interesting
questions: which inference types are most common, which models have the highest
performance on each reasoning type, and which types are the most challenging
for state of-the-art models? We hope that our annotations will enable more
fine-grained evaluation of models trained on ANLI, provide us with a deeper
understanding of where models fail and succeed, and help us determine how to
train better models in future.
