  We propose an unsupervised learning method to disentangle speech into content
representation and speaker identity representation. We apply this method to the
challenging one-shot cross-lingual voice conversion task to demonstrate the
effectiveness of the disentanglement. Inspired by $\beta$-VAE, we introduce a
learning objective that balances between the information captured by the
content and speaker representations. In addition, the inductive biases from the
architectural design and the training dataset further encourage the desired
disentanglement. Both objective and subjective evaluations show the
effectiveness of the proposed method in speech disentanglement and in one-shot
cross-lingual voice conversion.
