  Speech separation aims to separate individual voice from an audio mixture of
multiple simultaneous talkers. Although audio-only approaches achieve
satisfactory performance, they build on a strategy to handle the predefined
conditions, limiting their application in the complex auditory scene. Towards
the cocktail party problem, we propose a novel audio-visual speech separation
model. In our model, we use the face detector to detect the number of speakers
in the scene and use visual information to avoid the permutation problem. To
improve our model's generalization ability to unknown speakers, we extract
speech-related visual features from visual inputs explicitly by the
adversarially disentangled method, and use this feature to assist speech
separation. Besides, the time-domain approach is adopted, which could avoid the
phase reconstruction problem existing in the time-frequency domain models. To
compare our model's performance with other models, we create two benchmark
datasets of 2-speaker mixture from GRID and TCDTIMIT audio-visual datasets.
Through a series of experiments, our proposed model is shown to outperform the
state-of-the-art audio-only model and three audio-visual models.
