  Although supervised learning based on a deep neural network has recently
achieved substantial improvement on speech enhancement, the existing schemes
have either of two critical issues: spectrum or metric mismatches. The spectrum
mismatch is a well known issue that any spectrum modification after short-time
Fourier transform (STFT), in general, cannot be fully recovered after inverse
short-time Fourier transform (ISTFT). The metric mismatch is that a
conventional mean square error (MSE) loss function is typically sub-optimal to
maximize perceptual speech measure such as signal-to-distortion ratio (SDR),
perceptual evaluation of speech quality (PESQ) and short-time objective
intelligibility (STOI). This paper presents a new end-to-end denoising
framework. First, the network optimization is performed on the time-domain
signals after ISTFT to avoid the spectrum mismatch. Second, three loss
functions based on SDR, PESQ and STOI are proposed to minimize the metric
mismatch. The experimental result showed the proposed denoising scheme
significantly improved SDR, PESQ and STOI performance over the existing
methods. Moreover, the proposed scheme also provided good generalization
performance over generative denoising models on the perceptual speech metrics
not used as a loss function during training.
