  Conveying the linguistic content and maintaining the source speech's speaking
style, such as intonation and emotion, is essential in voice conversion (VC).
However, in a low-resource situation, where only limited utterances from the
target speaker are accessible, existing VC methods are hard to meet this
requirement and capture the target speaker's timber. In this work, a novel VC
model, referred to as MFC-StyleVC, is proposed for the low-resource VC task.
Specifically, speaker timbre constraint generated by clustering method is newly
proposed to guide target speaker timbre learning in different stages.
Meanwhile, to prevent over-fitting to the target speaker's limited data,
perceptual regularization constraints explicitly maintain model performance on
specific aspects, including speaking style, linguistic content, and speech
quality. Besides, a simulation mode is introduced to simulate the inference
process to alleviate the mismatch between training and inference. Extensive
experiments performed on highly expressive speech demonstrate the superiority
of the proposed method in low-resource VC.
