  Current voice conversion (VC) methods can successfully convert timbre of the
audio. As modeling source audio's prosody effectively is a challenging task,
there are still limitations of transferring source style to the converted
speech. This study proposes a source style transfer method based on
recognition-synthesis framework. Previously in speech generation task, prosody
can be modeled explicitly with prosodic features or implicitly with a latent
prosody extractor. In this paper, taking advantages of both, we model the
prosody in a hybrid manner, which effectively combines explicit and implicit
methods in a proposed prosody module. Specifically, prosodic features are used
to explicit model prosody, while VAE and reference encoder are used to
implicitly model prosody, which take Mel spectrum and bottleneck feature as
input respectively. Furthermore, adversarial training is introduced to remove
speaker-related information from the VAE outputs, avoiding leaking source
speaker information while transferring style. Finally, we use a modified
self-attention based encoder to extract sentential context from bottleneck
features, which also implicitly aggregates the prosodic aspects of source
speech from the layered representations. Experiments show that our approach is
superior to the baseline and a competitive system in terms of style transfer;
meanwhile, the speech quality and speaker similarity are well maintained.
