  State-of-the-art encoder-decoder models (e.g. for machine translation (MT) or
automatic speech recognition (ASR)) are constructed and trained end-to-end as
an atomic unit. No component of the model can be (re-)used without the others,
making it impossible to share parts, e.g. a high resourced decoder, across
tasks. We describe LegoNN, a procedure for building encoder-decoder
architectures in a way so that its parts can be applied to other tasks without
the need for any fine-tuning. To achieve this reusability, the interface
between encoder and decoder modules is grounded to a sequence of marginal
distributions over a pre-defined discrete vocabulary. We present two approaches
for ingesting these marginals; one is differentiable, allowing the flow of
gradients across the entire network, and the other is gradient-isolating. To
enable the portability of decoder modules between MT tasks for different source
languages and across other tasks like ASR, we introduce a modality agnostic
encoder which consists of a length control mechanism to dynamically adapt
encoders' output lengths in order to match the expected input length range of
pre-trained decoders. We present several experiments to demonstrate the
effectiveness of LegoNN models: a trained language generation LegoNN decoder
module from German-English (De-En) MT task can be reused without any
fine-tuning for the Europarl English ASR and the Romanian-English (Ro-En) MT
tasks, matching or beating the performance of baseline. After fine-tuning,
LegoNN models improve the Ro-En MT task by 1.5 BLEU points and achieve 12.5%
relative WER reduction on the Europarl ASR task. To show how the approach
generalizes, we compose a LegoNN ASR model from three modules -- each has been
learned within different end-to-end trained models on three different datasets
-- achieving an overall WER reduction of 19.5%.
