  Background noise, interfering speech and room reverberation frequently
distort target speech in real listening environments. In this study, we address
joint speech separation and dereverberation, which aims to separate target
speech from background noise, interfering speech and room reverberation. In
order to tackle this fundamentally difficult problem, we propose a novel
multimodal network that exploits both audio and visual signals. The proposed
network architecture adopts a two-stage strategy, where a separation module is
employed to attenuate background noise and interfering speech in the first
stage and a dereverberation module to suppress room reverberation in the second
stage. The two modules are first trained separately, and then integrated for
joint training, which is based on a new multi-objective loss function. Our
experimental results show that the proposed multimodal network yields
consistently better objective intelligibility and perceptual quality than
several one-stage and two-stage baselines. We find that our network achieves a
21.10% improvement in ESTOI and a 0.79 improvement in PESQ over the unprocessed
mixtures. Moreover, our network architecture does not require the knowledge of
the number of speakers.
