  End-to-end (E2E) automatic speech recognition (ASR) can operate in two modes:
streaming and non-streaming, each with its pros and cons. Streaming ASR
processes the speech frames in real-time as it is being received, while
non-streaming ASR waits for the entire speech utterance; thus, professionals
may have to operate in either mode to satisfy their application. In this work,
we present joint optimization of streaming and non-streaming ASR based on
multi-decoder and knowledge distillation. Primarily, we study 1) the encoder
integration of these ASR modules, followed by 2) separate decoders to make the
switching mode flexible, and enhancing performance by 3) incorporating
similarity-preserving knowledge distillation between the two modular encoders
and decoders. Evaluation results show 2.6%-5.3% relative character error rate
reductions (CERR) on CSJ for streaming ASR, and 8.3%-9.7% relative CERRs for
non-streaming ASR within a single model compared to multiple standalone
modules.
