  Commonsense reasoning is an appealing topic in natural language processing
(NLP) as it plays a fundamental role in supporting the human-like actions of
NLP systems. With large-scale language models as the backbone, unsupervised
pre-training on numerous corpora shows the potential to capture commonsense
knowledge. Current pre-trained language model (PLM)-based reasoning follows the
traditional practice using perplexity metric. However, commonsense reasoning is
more than existing probability evaluation, which is biased by word frequency.
This paper reconsiders the nature of commonsense reasoning and proposes a novel
commonsense reasoning metric, Non-Replacement Confidence (NRC). In detail, it
works on PLMs according to the Replaced Token Detection (RTD) pre-training
objective in ELECTRA, in which the corruption detection objective reflects the
confidence on contextual integrity that is more relevant to commonsense
reasoning than existing probability. Our proposed novel method boosts zero-shot
performance on two commonsense reasoning benchmark datasets and further seven
commonsense question-answering datasets. Our analysis shows that pre-endowed
commonsense knowledge, especially for RTD-based PLMs, is essential in
downstream reasoning.
