  We propose a highly controllable voice manipulation system that can perform
any-to-any voice conversion (VC) and prosody modulation simultaneously.
State-of-the-art VC systems can transfer sentence-level characteristics such as
speaker, emotion, and speaking style. However, manipulating the frame-level
prosody, such as pitch, energy and speaking rate, still remains challenging.
Our proposed model utilizes a frame-level prosody feature to effectively
transfer such properties. Specifically, pitch and energy trajectories are
integrated in a prosody conditioning module and then fed alongside speaker and
contents embeddings to a diffusion-based decoder generating a converted speech
mel-spectrogram. To adjust the speaking rate, our system includes a
self-supervised model based post-processing step which allows improved
controllability. The proposed model showed comparable speech quality and
improved intelligibility compared to a SOTA approach. It can cover a varying
range of fundamental frequency (F0), energy and speed modulation while
maintaining converted speech quality.
