  Expressive voice conversion (VC) conducts speaker identity conversion for
emotional speakers by jointly converting speaker identity and emotional style.
Emotional style modeling for arbitrary speakers in expressive VC has not been
extensively explored. Previous approaches have relied on vocoders for speech
reconstruction, which makes speech quality heavily dependent on the performance
of vocoders. A major challenge of expressive VC lies in emotion prosody
modeling. To address these challenges, this paper proposes a fully end-to-end
expressive VC framework based on a conditional denoising diffusion
probabilistic model (DDPM). We utilize speech units derived from
self-supervised speech models as content conditioning, along with deep features
extracted from speech emotion recognition and speaker verification systems to
model emotional style and speaker identity. Objective and subjective
evaluations show the effectiveness of our framework. Codes and samples are
publicly available.
