  Common intermediate language representation in neural machine translation can
be used to extend bilingual to multilingual systems by incremental training. In
this paper, we propose a new architecture based on introducing an interlingual
loss as an additional training objective. By adding and forcing this
interlingual loss, we are able to train multiple encoders and decoders for each
language, sharing a common intermediate representation. Translation results on
the low-resourced tasks (Turkish-English and Kazakh-English tasks, from the
popular Workshop on Machine Translation benchmark) show the following BLEU
improvements up to 2.8. However, results on a larger dataset (Russian-English
and Kazakh-English, from the same baselines) show BLEU loses if the same
amount. While our system is only providing improvements for the low-resourced
tasks in terms of translation quality, our system is capable of quickly
deploying new language pairs without retraining the rest of the system, which
may be a game-changer in some situations (i.e. in a disaster crisis where
international help is required towards a small region or to develop some
translation system for a client). Precisely, what is most relevant from our
architecture is that it is capable of: (1) reducing the number of production
systems, with respect to the number of languages, from quadratic to linear (2)
incrementally adding a new language in the system without retraining languages
previously there and (3) allowing for translations from the new language to all
the others present in the system
