  This paper presents a cross-lingual voice conversion framework that adopts a
modularized neural network. The modularized neural network has a common input
structure that is shared for both languages, and two separate output modules,
one for each language. The idea is motivated by the fact that phonetic systems
of languages are similar because humans share a common vocal production system,
but acoustic renderings, such as prosody and phonotactic, vary a lot from
language to language. The modularized neural network is trained to map Phonetic
PosteriorGram (PPG) to acoustic features for multiple speakers. It is
conditioned on a speaker i-vector to generate the desired target voice. We
validated the idea between English and Mandarin languages in objective and
subjective tests. In addition, mixed-lingual PPG derived from a unified
English-Mandarin acoustic model is proposed to capture the linguistic
information from both languages. It is found that our proposed modularized
neural network significantly outperforms the baseline approaches in terms of
speech quality and speaker individuality, and mixed-lingual PPG representation
further improves the conversion performance.
