  Machine intelligence is increasingly being linked to claims about sentience,
language processing, and an ability to comprehend and transform natural
language into a range of stimuli. We systematically analyze the ability of
DALL-E 2 to capture 8 grammatical phenomena pertaining to compositionality that
are widely discussed in linguistics and pervasive in human language: binding
principles and coreference, passives, word order, coordination, comparatives,
negation, ellipsis, and structural ambiguity. Whereas young children routinely
master these phenomena, learning systematic mappings between syntax and
semantics, DALL-E 2 is unable to reliably infer meanings that are consistent
with the syntax. These results challenge recent claims concerning the capacity
of such systems to understand of human language. We make available the full set
of test materials as a benchmark for future testing.
