  Current end-to-end approaches to Spoken Language Translation (SLT) rely on
limited training resources, especially for multilingual settings. On the other
hand, Multilingual Neural Machine Translation (MultiNMT) approaches rely on
higher-quality and more massive data sets. Our proposed method extends a
MultiNMT architecture based on language-specific encoders-decoders to the task
of Multilingual SLT (MultiSLT). Our method entirely eliminates the dependency
from MultiSLT data and it is able to translate while training only on ASR and
MultiNMT data.
  Our experiments on four different languages show that coupling the speech
encoder to the MultiNMT architecture produces similar quality translations
compared to a bilingual baseline ($\pm 0.2$ BLEU) while effectively allowing
for zero-shot MultiSLT. Additionally, we propose using an Adapter module for
coupling the speech inputs. This Adapter module produces consistent
improvements up to +6 BLEU points on the proposed architecture and +1 BLEU
point on the end-to-end baseline.
