  Shortcut reasoning is an irrational process of inference, which degrades the
robustness of an NLP model. While a number of previous work has tackled the
identification of shortcut reasoning, there are still two major limitations:
(i) a method for quantifying the severity of the discovered shortcut reasoning
is not provided; (ii) certain types of shortcut reasoning may be missed. To
address these issues, we propose a novel method for identifying shortcut
reasoning. The proposed method quantifies the severity of the shortcut
reasoning by leveraging out-of-distribution data and does not make any
assumptions about the type of tokens triggering the shortcut reasoning. Our
experiments on Natural Language Inference and Sentiment Analysis demonstrate
that our framework successfully discovers known and unknown shortcut reasoning
in the previous work.
