  Strong representations of target speakers can help extract important
information about speakers and detect corresponding temporal regions in
multi-speaker conversations. In this study, we propose a neural architecture
that simultaneously extracts speaker representations consistent with the
speaker diarization objective and detects the presence of each speaker on a
frame-by-frame basis regardless of the number of speakers in a conversation. A
speaker representation (called z-vector) extractor and a time-speaker
contextualizer, implemented by a residual network and processing data in both
temporal and speaker dimensions, are integrated into a unified framework. Tests
on the CALLHOME corpus show that our model outperforms most of the methods
proposed so far. Evaluations in a more challenging case with simultaneous
speakers ranging from 2 to 7 show that our model achieves 6.4% to 30.9%
relative diarization error rate reductions over several typical baselines.
