  This paper presents a new network architecture called multi-head decoder for
end-to-end speech recognition as an extension of a multi-head attention model.
In the multi-head attention model, multiple attentions are calculated, and
then, they are integrated into a single attention. On the other hand, instead
of the integration in the attention level, our proposed method uses multiple
decoders for each attention and integrates their outputs to generate a final
output. Furthermore, in order to make each head to capture the different
modalities, different attention functions are used for each head, leading to
the improvement of the recognition performance with an ensemble effect. To
evaluate the effectiveness of our proposed method, we conduct an experimental
evaluation using Corpus of Spontaneous Japanese. Experimental results
demonstrate that our proposed method outperforms the conventional methods such
as location-based and multi-head attention models, and that it can capture
different speech/linguistic contexts within the attention-based encoder-decoder
framework.
