  Machine translation software has seen rapid progress in recent years due to
the advancement of deep neural networks. People routinely use machine
translation software in their daily lives, such as ordering food in a foreign
restaurant, receiving medical diagnosis and treatment from foreign doctors, and
reading international political news online. However, due to the complexity and
intractability of the underlying neural networks, modern machine translation
software is still far from robust and can produce poor or incorrect
translations; this can lead to misunderstanding, financial loss, threats to
personal safety and health, and political conflicts. To address this problem,
we introduce referentially transparent inputs (RTIs), a simple, widely
applicable methodology for validating machine translation software. A
referentially transparent input is a piece of text that should have similar
translations when used in different contexts. Our practical implementation,
Purity, detects when this property is broken by a translation. To evaluate RTI,
we use Purity to test Google Translate and Bing Microsoft Translator with 200
unlabeled sentences, which detected 123 and 142 erroneous translations with
high precision (79.3% and 78.3%). The translation errors are diverse, including
examples of under-translation, over-translation, word/phrase mistranslation,
incorrect modification, and unclear logic.
