  This paper presents Transcribe-to-Diarize, a new approach for neural speaker
diarization that uses an end-to-end (E2E) speaker-attributed automatic speech
recognition (SA-ASR). The E2E SA-ASR is a joint model that was recently
proposed for speaker counting, multi-talker speech recognition, and speaker
identification from monaural audio that contains overlapping speech. Although
the E2E SA-ASR model originally does not estimate any time-related information,
we show that the start and end times of each word can be estimated with
sufficient accuracy from the internal state of the E2E SA-ASR by adding a small
number of learnable parameters. Similar to the target-speaker voice activity
detection (TS-VAD)-based diarization method, the E2E SA-ASR model is applied to
estimate speech activity of each speaker while it has the advantages of (i)
handling unlimited number of speakers, (ii) leveraging linguistic information
for speaker diarization, and (iii) simultaneously generating speaker-attributed
transcriptions. Experimental results on the LibriCSS and AMI corpora show that
the proposed method achieves significantly better diarization error rate than
various existing speaker diarization methods when the number of speakers is
unknown, and achieves a comparable performance to TS-VAD when the number of
speakers is given in advance. The proposed method simultaneously generates
speaker-attributed transcription with state-of-the-art accuracy.
