  Compositionality, the phenomenon where the meaning of a phrase can be derived
from its constituent parts, is a hallmark of human language. At the same time,
many phrases are non-compositional, carrying a meaning beyond that of each part
in isolation. Representing both of these types of phrases is critical for
language understanding, but it is an open question whether modern language
models (LMs) learn to do so; in this work we examine this question. We first
formulate a problem of predicting the LM-internal representations of longer
phrases given those of their constituents. We find that the representation of a
parent phrase can be predicted with some accuracy given an affine
transformation of its children. While we would expect the predictive accuracy
to correlate with human judgments of semantic compositionality, we find this is
largely not the case, indicating that LMs may not accurately distinguish
between compositional and non-compositional phrases. We perform a variety of
analyses, shedding light on when different varieties of LMs do and do not
generate compositional representations, and discuss implications for future
modeling work.
