  In this paper, we present a Distribution-Preserving Voice Anonymization
technique, as our submission to the VoicePrivacy Challenge 2020. We observe
that the challenge baseline system generates fake X-vectors which are very
similar to each other, significantly more so than those extracted from organic
speakers. This difference arises from averaging many X-vectors from a pool of
speakers in the anonymization process, causing a loss of information. We
propose a new method to generate fake X-vectors which overcomes these
limitations by preserving the distributional properties of X-vectors and their
intra-similarity. We use population data to learn the properties of the
X-vector space, before fitting a generative model which we use to sample fake
X-vectors. We show how this approach generates X-vectors that more closely
follow the expected intra-similarity distribution of organic speaker X-vectors.
Our method can be easily integrated with others as the anonymization component
of the system and removes the need to distribute a pool of speakers to use
during the anonymization. Our approach leads to an increase in EER of up to
$19.4\%$ in males and $11.1\%$ in females in scenarios where enrollment and
trial utterances are anonymized versus the baseline solution, demonstrating the
diversity of our generated voices.
