  This paper proposes a novel lip-reading driven deep learning framework for
speech enhancement. The proposed approach leverages the complementary strengths
of both deep learning and analytical acoustic modelling (filtering based
approach) as compared to recently published, comparatively simpler benchmark
approaches that rely only on deep learning. The proposed audio-visual (AV)
speech enhancement framework operates at two levels. In the first level, a
novel deep learning-based lip-reading regression model is employed. In the
second level, lip-reading approximated clean-audio features are exploited,
using an enhanced, visually-derived Wiener filter (EVWF), for the clean audio
power spectrum estimation. Specifically, a stacked long-short-term memory
(LSTM) based lip-reading regression model is designed for clean audio features
estimation using only temporal visual features considering different number of
prior visual frames. For clean speech spectrum estimation, a new
filterbank-domain EVWF is formulated, which exploits estimated speech features.
The proposed EVWF is compared with conventional Spectral Subtraction and
Log-Minimum Mean-Square Error methods using both ideal AV mapping and LSTM
driven AV mapping. The potential of the proposed speech enhancement framework
is evaluated under different dynamic real-world commercially-motivated
scenarios (e.g. cafe, public transport, pedestrian area) at different SNR
levels (ranging from low to high SNRs) using benchmark Grid and ChiME3 corpora.
For objective testing, perceptual evaluation of speech quality is used to
evaluate the quality of restored speech. For subjective testing, the standard
mean-opinion-score method is used with inferential statistics. Comparative
simulation results demonstrate significant lip-reading and speech enhancement
improvement in terms of both speech quality and speech intelligibility.
