  Minimal sentence pairs are frequently used to analyze the behavior of
language models. It is often assumed that model behavior on contrastive pairs
is predictive of model behavior at large. We argue that two conditions are
necessary for this assumption to hold: First, a tested hypothesis should be
well-motivated, since experiments show that contrastive evaluation can lead to
false positives. Secondly, test data should be chosen such as to minimize
distributional discrepancy between evaluation time and deployment time. For a
good approximation of deployment-time decoding, we recommend that minimal pairs
are created based on machine-generated text, as opposed to human-written
references. We present a contrastive evaluation suite for English-German MT
that implements this recommendation.
