  Beyond the conventional voice conversion (VC) where the speaker information
is converted without altering the linguistic content, the background sounds are
informative and need to be retained in some real-world scenarios, such as VC in
movie/video and VC in music where the voice is entangled with background
sounds. As a new VC framework, we have developed a noisy-to-noisy (N2N) VC
framework to convert the speaker's identity while preserving the background
sounds. Although our framework consisting of a denoising module and a VC module
well handles the background sounds, the VC module is sensitive to the
distortion caused by the denoising module. To address this distortion issue, in
this paper we propose the improved VC module to directly model the noisy speech
waveform while controlling the background sounds. The experimental results have
demonstrated that our improved framework significantly outperforms the previous
one and achieves an acceptable score in terms of naturalness, while reaching
comparable similarity performance to the upper bound of our framework.
