  The multi-decoder (MD) end-to-end speech translation model has demonstrated
high translation quality by searching for better intermediate automatic speech
recognition (ASR) decoder states as hidden intermediates (HI). It is a two-pass
decoding model decomposing the overall task into ASR and machine translation
sub-tasks. However, the decoding speed is not fast enough for real-world
applications because it conducts beam search for both sub-tasks during
inference. We propose Fast-MD, a fast MD model that generates HI by
non-autoregressive (NAR) decoding based on connectionist temporal
classification (CTC) outputs followed by an ASR decoder. We investigated two
types of NAR HI: (1) parallel HI by using an autoregressive Transformer ASR
decoder and (2) masked HI by using Mask-CTC, which combines CTC and the
conditional masked language model. To reduce a mismatch in the ASR decoder
between teacher-forcing during training and conditioning on CTC outputs during
testing, we also propose sampling CTC outputs during training. Experimental
evaluations on three corpora show that Fast-MD achieved about 2x and 4x faster
decoding speed than that of the na\"ive MD model on GPU and CPU with comparable
translation quality. Adopting the Conformer encoder and intermediate CTC loss
further boosts its quality without sacrificing decoding speed.
