  Metamorphic testing has recently been used to check the safety of neural NLP
models. Its main advantage is that it does not rely on a ground truth to
generate test cases. However, existing studies are mostly concerned with
robustness-like metamorphic relations, limiting the scope of linguistic
properties they can test. We propose three new classes of metamorphic
relations, which address the properties of systematicity, compositionality and
transitivity. Unlike robustness, our relations are defined over multiple source
inputs, thus increasing the number of test cases that we can produce by a
polynomial factor. With them, we test the internal consistency of
state-of-the-art NLP models, and show that they do not always behave according
to their expected linguistic properties. Lastly, we introduce a novel graphical
notation that efficiently summarises the inner structure of metamorphic
relations.
