  In this paper, we aim to create weak alignment supervision from an existing
hybrid system to aid the end-to-end modeling of automatic speech recognition.
Towards this end, we use the existing hybrid ASR system to produce triphone
alignments of the training audios. We then create a cross-entropy loss at a
certain layer of the encoder using the derived alignments. In contrast to the
general one-hot cross-entropy losses, here we use a cross-entropy loss with a
label smoothing parameter to regularize the supervision. As a comparison, we
also conduct the experiments with one-hot cross-entropy losses and CTC losses
with loss weighting. The results show that placing the weak alignment
supervision with the label smoothing parameter of 0.5 at the third encoder
layer outperforms the other two approaches and leads to about 5\% relative WER
reduction on the TED-LIUM 2 dataset over the baseline. We see similar
improvements when applying the method out-of-the-box on a Tagalog end-to-end
ASR system.
