  Speaker extraction and diarization are two enabling techniques for real-world
speech applications. Speaker extraction aims to extract a target speaker's
voice from a speech mixture, while speaker diarization demarcates speech
segments by speaker, annotating `who spoke when'. Previous studies have
typically treated the two tasks independently. In practical applications, it is
more meaningful to have knowledge about `who spoke what and when', which is
captured by the two tasks. The two tasks share a similar objective of
disentangling speakers. Speaker extraction operates in the frequency domain,
whereas diarization is in the temporal domain. It is logical to believe that
speaker activities obtained from speaker diarization can benefit speaker
extraction, while the extracted speech offers more accurate speaker activity
detection than the speech mixture. In this paper, we propose a unified model
called Universal Speaker Extraction and Diarization (USED) to address output
inconsistency and scenario mismatch issues. It is designed to manage speech
mixture with varying overlap ratios and variable number of speakers. We show
that the USED model significantly outperforms the competitive baselines for
speaker extraction and diarization tasks on LibriMix and SparseLibriMix
datasets. We further validate the diarization performance on CALLHOME, a
dataset based on real recordings, and experimental results indicate that our
model surpasses recently proposed approaches.
