  Previous audio-visual speech separation methods use the synchronization of
the speaker's facial movement and speech in the video to supervise the speech
separation in a self-supervised way. In this paper, we propose a model to solve
the speech separation problem assisted by both face and sign language, which we
call the extended speech separation problem. We design a general deep learning
network for learning the combination of three modalities, audio, face, and sign
language information, for better solving the speech separation problem. To
train the model, we introduce a large-scale dataset named the Chinese Sign
Language News Speech (CSLNSpeech) dataset, in which three modalities of audio,
face, and sign language coexist. Experiment results show that the proposed
model has better performance and robustness than the usual audio-visual system.
Besides, sign language modality can also be used alone to supervise speech
separation tasks, and the introduction of sign language is helpful for
hearing-impaired people to learn and communicate. Last, our model is a general
speech separation framework and can achieve very competitive separation
performance on two open-source audio-visual datasets. The code is available at
https://github.com/iveveive/SLNSpeech
