  We present a novel approach to any-to-one (A2O) voice conversion (VC) in a
sequence-to-sequence (seq2seq) framework. A2O VC aims to convert any speaker,
including those unseen during training, to a fixed target speaker. We utilize
vq-wav2vec (VQW2V), a discretized self-supervised speech representation that
was learned from massive unlabeled data, which is assumed to be
speaker-independent and well corresponds to underlying linguistic contents.
Given a training dataset of the target speaker, we extract VQW2V and acoustic
features to estimate a seq2seq mapping function from the former to the latter.
With the help of a pretraining method and a newly designed postprocessing
technique, our model can be generalized to only 5 min of data, even
outperforming the same model trained with parallel data.
