  Recently, language models have demonstrated strong performance on various
natural language understanding tasks. Language models trained on large
human-generated corpus encode not only a significant amount of human knowledge,
but also the human stereotype. As more and more downstream tasks have
integrated language models as part of the pipeline, it is necessary to
understand the internal stereotypical representation in order to design the
methods for mitigating the negative effects. In this paper, we use
counterexamples to examine the internal stereotypical knowledge in pre-trained
language models (PLMs) that can lead to stereotypical preference. We mainly
focus on gender stereotypes, but the method can be extended to other types of
stereotype. We evaluate 7 PLMs on 9 types of cloze-style prompt with different
information and base knowledge. The results indicate that PLMs show a certain
amount of robustness against unrelated information and preference of shallow
linguistic cues, such as word position and syntactic structure, but a lack of
interpreting information by meaning. Such findings shed light on how to
interact with PLMs in a neutral approach for both finetuning and evaluation.
