  Despite the significant improvements in speaker recognition enabled by deep
neural networks, unsatisfactory performance persists under noisy environments.
In this paper, we train the speaker embedding network to learn the "clean"
embedding of the noisy utterance. Specifically, the network is trained with the
original speaker identification loss with an auxiliary within-sample
variability-invariant loss. This auxiliary variability-invariant loss is used
to learn the same embedding among the clean utterance and its noisy copies and
prevents the network from encoding the undesired noises or variabilities into
the speaker representation. Furthermore, we investigate the data preparation
strategy for generating clean and noisy utterance pairs on-the-fly. The
strategy generates different noisy copies for the same clean utterance at each
training step, helping the speaker embedding network generalize better under
noisy environments. Experiments on VoxCeleb1 indicate that the proposed
training framework improves the performance of the speaker verification system
in both clean and noisy conditions.
