  The state-of-the-art speaker diarization systems use agglomerative
hierarchical clustering (AHC) which performs the clustering of previously
learned neural embeddings. While the clustering approach attempts to identify
speaker clusters, the AHC algorithm does not involve any further learning. In
this paper, we propose a novel algorithm for hierarchical clustering which
combines the speaker clustering along with a representation learning framework.
The proposed approach is based on principles of self-supervised learning where
the self-supervision is derived from the clustering algorithm. The
representation learning network is trained with a regularized triplet loss
using the clustering solution at the current step while the clustering
algorithm uses the deep embeddings from the representation learning step. By
combining the self-supervision based representation learning along with the
clustering algorithm, we show that the proposed algorithm improves
significantly 29% relative improvement) over the AHC algorithm with cosine
similarity for a speaker diarization task on CALLHOME dataset. In addition, the
proposed approach also improves over the state-of-the-art system with PLDA
affinity matrix with 10% relative improvement in DER.
