  Modern speaker recognition system relies on abundant and balanced datasets
for classification training. However, diverse defective datasets, such as
partially-labelled, small-scale, and imbalanced datasets, are common in
real-world applications. Previous works usually studied specific solutions for
each scenario from the algorithm perspective. However, the root cause of these
problems lies in dataset imperfections. To address these challenges with a
unified solution, we propose the Voice Conversion Augmentation (VCA) strategy
to obtain pseudo speech from the training set. Furthermore, to guarantee
generation quality, we designed the VCA-NN~(nearest neighbours) strategy to
select source speech from utterances that are close to the target speech in the
representation space. Our experimental results on three created datasets
demonstrated that VCA-NN effectively mitigates these dataset problems, which
provides a new direction for handling the speaker recognition problems from the
data aspect.
