  We have developed a diffusion-based speech refiner that improves the
reference-free perceptual quality of the audio predicted by preceding
single-channel speech separation models. Although modern deep neural
network-based speech separation models have show high performance in
reference-based metrics, they often produce perceptually unnatural artifacts.
The recent advancements made to diffusion models motivated us to tackle this
problem by restoring the degraded parts of initial separations with a
generative approach. Utilizing the denoising diffusion restoration model (DDRM)
as a basis, we propose a shared DDRM-based refiner that generates samples
conditioned on the global information of preceding outputs from arbitrary
speech separation models. We experimentally show that our refiner can provide a
clearer harmonic structure of speech and improves the reference-free metric of
perceptual quality for arbitrary preceding model architectures. Furthermore, we
tune the variance of the measurement noise based on preceding outputs, which
results in higher scores in both reference-free and reference-based metrics.
The separation quality can also be further improved by blending the
discriminative and generative outputs.
