  State-of-the-art approaches to speaker anonymization typically employ some
form of perturbation function to conceal speaker information contained within
an x-vector embedding, then resynthesize utterances in the voice of a new
pseudo-speaker using a vocoder. Strategies to improve the x-vector
anonymization function have attracted considerable research effort, whereas
vocoder impacts are generally neglected. In this paper, we show that the impact
of the vocoder is substantial and sometimes dominant. The vocoder drift, namely
the difference between the x-vector vocoder input and that which can be
extracted subsequently from the output, is learnable and can hence be reversed
by an attacker; anonymization can be undone and the level of privacy protection
provided by such approaches might be weaker than previously thought. The
findings call into question the focus upon x-vector anonymization, prompting
the need for greater attention to vocoder impacts and stronger attack models
alike.
