  Recently, there has been growing interest in multi-speaker speech
recognition, where the utterances of multiple speakers are recognized from
their mixture. Promising techniques have been proposed for this task, but
earlier works have required additional training data such as isolated source
signals or senone alignments for effective learning. In this paper, we propose
a new sequence-to-sequence framework to directly decode multiple label
sequences from a single speech sequence by unifying source separation and
speech recognition functions in an end-to-end manner. We further propose a new
objective function to improve the contrast between the hidden vectors to avoid
generating similar hypotheses. Experimental results show that the model is
directly able to learn a mapping from a speech mixture to multiple label
sequences, achieving 83.1 % relative improvement compared to a model trained
without the proposed objective. Interestingly, the results are comparable to
those produced by previous end-to-end works featuring explicit separation and
recognition modules.
