  The Winograd Schema (WS) has been proposed as a test for measuring
commonsense capabilities of models. Recently, pre-trained language model-based
approaches have boosted performance on some WS benchmarks but the source of
improvement is still not clear. This paper suggests that the apparent progress
on WS may not necessarily reflect progress in commonsense reasoning. To support
this claim, we first show that the current evaluation method of WS is
sub-optimal and propose a modification that uses twin sentences for evaluation.
We also propose two new baselines that indicate the existence of artifacts in
WS benchmarks. We then develop a method for evaluating WS-like sentences in a
zero-shot setting to account for the commonsense reasoning abilities acquired
during the pretraining and observe that popular language models perform
randomly in this setting when using our more strict evaluation. We conclude
that the observed progress is mostly due to the use of supervision in training
WS models, which is not likely to successfully support all the required
commonsense reasoning skills and knowledge.
