  When we use End-to-end automatic speech recognition (E2E-ASR) system for
real-world applications, a voice activity detection (VAD) system is usually
needed to improve the performance and to reduce the computational cost by
discarding non-speech parts in the audio. Usually ASR and VAD systems are
trained and utilized independently to each other. In this paper, we present a
novel multi-task learning (MTL) framework that incorporates VAD into the ASR
system. The proposed system learns ASR and VAD jointly in the training stage.
With the assistance of VAD, the ASR performance improves as its connectionist
temporal classification (CTC) loss function can leverage the VAD alignment
information. In the inference stage, the proposed system removes non-speech
parts at low computational cost and recognizes speech parts with high
robustness. Experimental results on segmented speech data show that by
utilizing VAD information, the proposed method outperforms the baseline ASR
system on both English and Chinese datasets. On unsegmented speech data, we
find that the system outperforms the ASR systems that build an extra GMM-based
or DNN-based voice activity detector.
