  Compositionality is a widely discussed property of natural languages,
although its exact definition has been elusive. We focus on the proposal that
compositionality can be assessed by measuring meaning-form correlation. We
analyze meaning-form correlation on three sets of languages: (i) artificial toy
languages tailored to be compositional, (ii) a set of English dictionary
definitions, and (iii) a set of English sentences drawn from literature. We
find that linguistic phenomena such as synonymy and ungrounded stop-words weigh
on MFC measurements, and that straightforward methods to mitigate their effects
have widely varying results depending on the dataset they are applied to. Data
and code are made publicly available.
