  We detect out-of-training-distribution sentences in Neural Machine
Translation using the Bayesian Deep Learning equivalent of Transformer models.
For this we develop a new measure of uncertainty designed specifically for long
sequences of discrete random variables -- i.e. words in the output sentence.
Our new measure of uncertainty solves a major intractability in the naive
application of existing approaches on long sentences. We use our new measure on
a Transformer model trained with dropout approximate inference. On the task of
German-English translation using WMT13 and Europarl, we show that with dropout
uncertainty our measure is able to identify when Dutch source sentences,
sentences which use the same word types as German, are given to the model
instead of German.
