  Pretrained language models, such as BERT and RoBERTa, have shown large
improvements in the commonsense reasoning benchmark COPA. However, recent work
found that many improvements in benchmarks of natural language understanding
are not due to models learning the task, but due to their increasing ability to
exploit superficial cues, such as tokens that occur more often in the correct
answer than the wrong one. Are BERT's and RoBERTa's good performance on COPA
also caused by this? We find superficial cues in COPA, as well as evidence that
BERT exploits these cues. To remedy this problem, we introduce Balanced COPA,
an extension of COPA that does not suffer from easy-to-exploit single token
cues. We analyze BERT's and RoBERTa's performance on original and Balanced
COPA, finding that BERT relies on superficial cues when they are present, but
still achieves comparable performance once they are made ineffective,
suggesting that BERT learns the task to a certain degree when forced to. In
contrast, RoBERTa does not appear to rely on superficial cues.
