  Self-Supervised Learning (SSL) has made great strides recently. SSL speech
models achieve decent performance on a wide range of downstream tasks,
suggesting that they extract different aspects of information from speech.
However, how SSL models store various information in hidden representations
without interfering is still poorly understood. Taking the recently successful
SSL model, HuBERT, as an example, we explore how the SSL model processes and
stores speaker information in the representation. We found that HuBERT stores
speaker information in representations whose positions correspond to silences
in a waveform. There are several pieces of evidence. (1) We find that the
utterances with more silent parts in the waveforms have better Speaker
Identification (SID) accuracy. (2) If we use the whole utterances for SID, the
silence part always contributes more to the SID task. (3) If we only use the
representation of a part of the utterance for SID, the silenced part has higher
accuracy than the other parts. Our findings not only contribute to a better
understanding of SSL models but also improve performance. By simply adding
silence to the original waveform, HuBERT improved its accuracy on SID by nearly
2%.
