  The widespread adoption of speech-based online services raises security and
privacy concerns regarding the data that they use and share. If the data were
compromised, attackers could exploit user speech to bypass speaker verification
systems or even impersonate users. To mitigate this, we propose DeID-VC, a
speaker de-identification system that converts a real speaker to pseudo
speakers, thus removing or obfuscating the speaker-dependent attributes from a
spoken voice. The key components of DeID-VC include a Variational Autoencoder
(VAE) based Pseudo Speaker Generator (PSG) and a voice conversion Autoencoder
(AE) under zero-shot settings. With the help of PSG, DeID-VC can assign unique
pseudo speakers at speaker level or even at utterance level. Also, two novel
learning objectives are added to bridge the gap between training and inference
of zero-shot voice conversion. We present our experimental results with word
error rate (WER) and equal error rate (EER), along with three subjective
metrics to evaluate the generated output of DeID-VC. The result shows that our
method substantially improved intelligibility (WER 10% lower) and
de-identification effectiveness (EER 5% higher) compared to our baseline. Code
and listening demo: https://github.com/a43992899/DeID-VC
