  Recent studies of the emergent capabilities of transformer-based Natural
Language Understanding (NLU) models have indicated that they have an
understanding of lexical and compositional semantics. We provide evidence that
suggests these claims should be taken with a grain of salt: we find that
state-of-the-art Natural Language Inference (NLI) models are sensitive towards
minor semantics preserving surface-form variations, which lead to sizable
inconsistent model decisions during inference. Notably, this behaviour differs
from valid and in-depth comprehension of compositional semantics, however does
neither emerge when evaluating model accuracy on standard benchmarks nor when
probing for syntactic, monotonic, and logically robust reasoning. We propose a
novel framework to measure the extent of semantic sensitivity. To this end, we
evaluate NLI models on adversarially generated examples containing minor
semantics-preserving surface-form input noise. This is achieved using
conditional text generation, with the explicit condition that the NLI model
predicts the relationship between the original and adversarial inputs as a
symmetric equivalence entailment. We systematically study the effects of the
phenomenon across NLI models for $\textbf{in-}$ and $\textbf{out-of-}$ domain
settings. Our experiments show that semantic sensitivity causes performance
degradations of $12.92\%$ and $23.71\%$ average over $\textbf{in-}$ and
$\textbf{out-of-}$ domain settings, respectively. We further perform ablation
studies, analysing this phenomenon across models, datasets, and variations in
inference and show that semantic sensitivity can lead to major inconsistency
within model predictions.
