  Individuals with hearing impairments face challenges in their ability to
comprehend speech, particularly in noisy environments. The aim of this study is
to explore the effectiveness of audio-visual speech enhancement (AVSE) in
enhancing the intelligibility of vocoded speech in cochlear implant (CI)
simulations. Notably, the study focuses on a challenged scenario where there is
limited availability of training data for the AVSE task. To address this
problem, we propose a novel deep neural network framework termed
Self-Supervised Learning-based AVSE (SSL-AVSE). The proposed SSL-AVSE combines
visual cues, such as lip and mouth movements, from the target speakers with
corresponding audio signals. The contextually combined audio and visual data
are then fed into a Transformer-based SSL AV-HuBERT model to extract features,
which are further processed using a BLSTM-based SE model. The results
demonstrate several key findings. Firstly, SSL-AVSE successfully overcomes the
issue of limited data by leveraging the AV-HuBERT model. Secondly, by
fine-tuning the AV-HuBERT model parameters for the target SE task, significant
performance improvements are achieved. Specifically, there is a notable
enhancement in PESQ (Perceptual Evaluation of Speech Quality) from 1.43 to 1.67
and in STOI (Short-Time Objective Intelligibility) from 0.70 to 0.74.
Furthermore, the performance of the SSL-AVSE was evaluated using CI vocoded
speech to assess the intelligibility for CI users. Comparative experimental
outcomes reveal that in the presence of dynamic noises encountered during human
conversations, SSL-AVSE exhibits a substantial improvement. The NCM (Normal
Correlation Matrix) values indicate an increase of 26.5% to 87.2% compared to
the noisy baseline.
