  While standard speaker diarization attempts to answer the question "who
spoken when", most of relevant applications in reality are more interested in
determining "who spoken what". Whether it is the conventional modularized
approach or the more recent end-to-end neural diarization (EEND), an additional
automatic speech recognition (ASR) model and an orchestration algorithm are
required to associate the speaker labels with recognized words. In this paper,
we propose Word-level End-to-End Neural Diarization (WEEND) with auxiliary
network, a multi-task learning algorithm that performs end-to-end ASR and
speaker diarization in the same neural architecture. That is, while speech is
being recognized, speaker labels are predicted simultaneously for each
recognized word. Experimental results demonstrate that WEEND outperforms the
turn-based diarization baseline system on all 2-speaker short-form scenarios
and has the capability to generalize to audio lengths of 5 minutes. Although
3+speaker conversations are harder, we find that with enough in-domain training
data, WEEND has the potential to deliver high quality diarized text.
