  We study a novel neural architecture and its training strategies of speaker
encoder for speaker recognition without using any identity labels. The speaker
encoder is trained to extract a fixed-size speaker embedding from a spoken
utterance of various length. Contrastive learning is a typical self-supervised
learning technique. However, the quality of the speaker encoder depends very
much on the sampling strategy of positive and negative pairs. It is common that
we sample a positive pair of segments from the same utterance. Unfortunately,
such poor-man's positive pairs (PPP) lack necessary diversity for the training
of a robust encoder. In this work, we propose a multi-modal contrastive
learning technique with novel sampling strategies. By cross-referencing between
speech and face data, we study a method that finds diverse positive pairs (DPP)
for contrastive learning, thus improving the robustness of the speaker encoder.
We train the speaker encoder on the VoxCeleb2 dataset without any speaker
labels, and achieve an equal error rate (EER) of 2.89\%, 3.17\% and 6.27\%
under the proposed progressive clustering strategy, and an EER of 1.44\%,
1.77\% and 3.27\% under the two-stage learning strategy with pseudo labels, on
the three test sets of VoxCeleb1. This novel solution outperforms the
state-of-the-art self-supervised learning methods by a large margin, at the
same time, achieves comparable results with the supervised learning
counterpart. We also evaluate our self-supervised learning technique on LRS2
and LRW datasets, where the speaker information is unknown. All experiments
suggest that the proposed neural architecture and sampling strategies are
robust across datasets.
