  Any-to-any voice conversion technologies convert the vocal timbre of an
utterance to any speaker even unseen during training. Although there have been
several state-of-the-art any-to-any voice conversion models, they were all
based on clean utterances to convert successfully. However, in real-world
scenarios, it is difficult to collect clean utterances of a speaker, and they
are usually degraded by noises or reverberations. It thus becomes highly
desired to understand how these degradations affect voice conversion and build
a degradation-robust model. We report in this paper the first comprehensive
study on the degradation robustness of any-to-any voice conversion. We show
that the performance of state-of-the-art models nowadays was severely hampered
given degraded utterances. To this end, we then propose speech enhancement
concatenation and denoising training to improve the robustness. In addition to
common degradations, we also consider adversarial noises, which alter the model
output significantly yet are human-imperceptible. It was shown that both
concatenations with off-the-shelf speech enhancement models and denoising
training on voice conversion models could improve the robustness, while each of
them had pros and cons.
