  A recent line of work in NLP focuses on the (dis)ability of models to
generalise compositionally for artificial languages. However, when considering
natural language tasks, the data involved is not strictly, or locally,
compositional. Quantifying the compositionality of data is a challenging task,
which has been investigated primarily for short utterances. We use recursive
neural models (Tree-LSTMs) with bottlenecks that limit the transfer of
information between nodes. We illustrate that comparing data's representations
in models with and without the bottleneck can be used to produce a
compositionality metric. The procedure is applied to the evaluation of
arithmetic expressions using synthetic data, and sentiment classification using
natural language data. We demonstrate that compression through a bottleneck
impacts non-compositional examples disproportionately and then use the
bottleneck compositionality metric (BCM) to distinguish compositional from
non-compositional samples, yielding a compositionality ranking over a dataset.
