  In this paper, we present the first comprehensive categorization of essential
commonsense knowledge for answering the Winograd Schema Challenge (WSC). For
each of the questions, we invite annotators to first provide reasons for making
correct decisions and then categorize them into six major knowledge categories.
By doing so, we better understand the limitation of existing methods (i.e.,
what kind of knowledge cannot be effectively represented or inferred with
existing methods) and shed some light on the commonsense knowledge that we need
to acquire in the future for better commonsense reasoning. Moreover, to
investigate whether current WSC models can understand the commonsense or they
simply solve the WSC questions based on the statistical bias of the dataset, we
leverage the collected reasons to develop a new task called WinoWhy, which
requires models to distinguish plausible reasons from very similar but wrong
reasons for all WSC questions. Experimental results prove that even though
pre-trained language representation models have achieved promising progress on
the original WSC dataset, they are still struggling at WinoWhy. Further
experiments show that even though supervised models can achieve better
performance, the performance of these models can be sensitive to the dataset
distribution. WinoWhy and all codes are available at:
https://github.com/HKUST-KnowComp/WinoWhy.
