  A characteristic feature of human semantic cognition is its ability to not
only store and retrieve the properties of concepts observed through experience,
but to also facilitate the inheritance of properties (can breathe) from
superordinate concepts (animal) to their subordinates (dog) -- i.e. demonstrate
property inheritance. In this paper, we present COMPS, a collection of minimal
pair sentences that jointly tests pre-trained language models (PLMs) on their
ability to attribute properties to concepts and their ability to demonstrate
property inheritance behavior. Analyses of 22 different PLMs on COMPS reveal
that they can easily distinguish between concepts on the basis of a property
when they are trivially different, but find it relatively difficult when
concepts are related on the basis of nuanced knowledge representations.
Furthermore, we find that PLMs can demonstrate behavior consistent with
property inheritance to a great extent, but fail in the presence of distracting
information, which decreases the performance of many models, sometimes even
below chance. This lack of robustness in demonstrating simple reasoning raises
important questions about PLMs' capacity to make correct inferences even when
they appear to possess the prerequisite knowledge.
