  End-to-end approaches for sequence tasks are becoming increasingly popular.
Yet for complex sequence tasks, like speech translation, systems that cascade
several models trained on sub-tasks have shown to be superior, suggesting that
the compositionality of cascaded systems simplifies learning and enables
sophisticated search capabilities. In this work, we present an end-to-end
framework that exploits compositionality to learn searchable hidden
representations at intermediate stages of a sequence model using decomposed
sub-tasks. These hidden intermediates can be improved using beam search to
enhance the overall performance and can also incorporate external models at
intermediate stages of the network to re-score or adapt towards out-of-domain
data. One instance of the proposed framework is a Multi-Decoder model for
speech translation that extracts the searchable hidden intermediates from a
speech recognition sub-task. The model demonstrates the aforementioned benefits
and outperforms the previous state-of-the-art by around +6 and +3 BLEU on the
two test sets of Fisher-CallHome and by around +3 and +4 BLEU on the
English-German and English-French test sets of MuST-C.
