  Human speech processing is inherently multimodal, where visual cues (lip
movements) help to better understand the speech in noise. Lip-reading driven
speech enhancement significantly outperforms benchmark audio-only approaches at
low signal-to-noise ratios (SNRs). However, at high SNRs or low levels of
background noise, visual cues become fairly less effective for speech
enhancement. Therefore, a more optimal, context-aware audio-visual (AV) system
is required, that contextually utilises both visual and noisy audio features
and effectively accounts for different noisy conditions. In this paper, we
introduce a novel contextual AV switching component that contextually exploits
AV cues with respect to different operating conditions to estimate clean audio,
without requiring any SNR estimation. The switching module switches between
visual-only (V-only), audio-only (A-only), and both AV cues at low, high and
moderate SNR levels, respectively. The contextual AV switching component is
developed by integrating a convolutional neural network and long-short-term
memory network. For testing, the estimated clean audio features are utilised by
the developed novel enhanced visually derived Wiener filter for clean audio
power spectrum estimation. The contextual AV speech enhancement method is
evaluated under real-world scenarios using benchmark Grid and ChiME3 corpora.
For objective testing, perceptual evaluation of speech quality is used to
evaluate the quality of the restored speech. For subjective testing, the
standard mean-opinion-score method is used. The critical analysis and
comparative study demonstrate the outperformance of proposed contextual AV
approach, over A-only, V-only, spectral subtraction, and log-minimum mean
square error based speech enhancement methods at both low and high SNRs,
revealing its capability to tackle spectro-temporal variation in any real-world
noisy condition.
