  Obtaining human-like performance in NLP is often argued to require
compositional generalisation. Whether neural networks exhibit this ability is
usually studied by training models on highly compositional synthetic data.
However, compositionality in natural language is much more complex than the
rigid, arithmetic-like version such data adheres to, and artificial
compositionality tests thus do not allow us to determine how neural models deal
with more realistic forms of compositionality. In this work, we re-instantiate
three compositionality tests from the literature and reformulate them for
neural machine translation (NMT). Our results highlight that: i) unfavourably,
models trained on more data are more compositional; ii) models are sometimes
less compositional than expected, but sometimes more, exemplifying that
different levels of compositionality are required, and models are not always
able to modulate between them correctly; iii) some of the non-compositional
behaviours are mistakes, whereas others reflect the natural variation in data.
Apart from an empirical study, our work is a call to action: we should rethink
the evaluation of compositionality in neural networks and develop benchmarks
using real data to evaluate compositionality on natural language, where
composing meaning is not as straightforward as doing the math.
