  Natural Language Inference (NLI) has been a cornerstone task in evaluating
language models' inferential reasoning capabilities. However, the standard
three-way classification scheme used in NLI has well-known shortcomings in
evaluating models' ability to capture the nuances of natural human reasoning.
In this paper, we argue that the operationalization of the neutral label in
current NLI datasets has low validity, is interpreted inconsistently, and that
at least one important sense of neutrality is often ignored. We uncover the
detrimental impact of these shortcomings, which in some cases leads to
annotation datasets that actually decrease performance on downstream tasks. We
compare approaches of handling annotator disagreement and identify flaws in a
recent NLI dataset that designs an annotator study based on a problematic
operationalization. Our findings highlight the need for a more refined
evaluation framework for NLI, and we hope to spark further discussion and
action in the NLP community.
