  Recently, there has been an increasing interest in end-to-end speech
recognition that directly transcribes speech to text without any predefined
alignments. One approach is the attention-based encoder-decoder framework that
learns a mapping between variable-length input and output sequences in one step
using a purely data-driven method. The attention model has often been shown to
improve the performance over another end-to-end approach, the Connectionist
Temporal Classification (CTC), mainly because it explicitly uses the history of
the target character without any conditional independence assumptions. However,
we observed that the performance of the attention has shown poor results in
noisy condition and is hard to learn in the initial training stage with long
input sequences. This is because the attention model is too flexible to predict
proper alignments in such cases due to the lack of left-to-right constraints as
used in CTC. This paper presents a novel method for end-to-end speech
recognition to improve robustness and achieve fast convergence by using a joint
CTC-attention model within the multi-task learning framework, thereby
mitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks
demonstrates its advantages over both the CTC and attention-based
encoder-decoder baselines, showing 5.4-14.6% relative improvements in Character
Error Rate (CER).
