  Recent studies have increasingly acknowledged the advantages of incorporating
visual data into speech enhancement (SE) systems. In this paper, we introduce a
novel audio-visual SE approach, termed DCUC-Net (deep complex U-Net with
conformer network). The proposed DCUC-Net leverages complex domain features and
a stack of conformer blocks. The encoder and decoder of DCUC-Net are designed
using a complex U-Net-based framework. The audio and visual signals are
processed using a complex encoder and a ResNet-18 model, respectively. These
processed signals are then fused using the conformer blocks and transformed
into enhanced speech waveforms via a complex decoder. The conformer blocks
consist of a combination of self-attention mechanisms and convolutional
operations, enabling DCUC-Net to effectively capture both global and local
audio-visual dependencies. Our experimental results demonstrate the
effectiveness of DCUC-Net, as it outperforms the baseline model from the
COG-MHEAR AVSE Challenge 2023 by a notable margin of 0.14 in terms of PESQ.
Additionally, the proposed DCUC-Net performs comparably to a state-of-the-art
model and outperforms all other compared models on the Taiwan Mandarin speech
with video (TMSV) dataset.
