  Understanding longer narratives or participating in conversations requires
tracking of discourse entities that have been mentioned. Indefinite noun
phrases (NPs), such as 'a dog', frequently introduce discourse entities but
this behavior is modulated by sentential operators such as negation. For
example, 'a dog' in 'Arthur doesn't own a dog' does not introduce a discourse
entity due to the presence of negation. In this work, we adapt the
psycholinguistic assessment of language models paradigm to higher-level
linguistic phenomena and introduce an English evaluation suite that targets the
knowledge of the interactions between sentential operators and indefinite NPs.
We use this evaluation suite for a fine-grained investigation of the entity
tracking abilities of the Transformer-based models GPT-2 and GPT-3. We find
that while the models are to a certain extent sensitive to the interactions we
investigate, they are all challenged by the presence of multiple NPs and their
behavior is not systematic, which suggests that even models at the scale of
GPT-3 do not fully acquire basic entity tracking abilities.
