  While existing Audio-Visual Speech Separation (AVSS) methods primarily
concentrate on the audio-visual fusion strategy for two-speaker separation,
they demonstrate a severe performance drop in the multi-speaker separation
scenarios. Typically, AVSS methods employ guiding videos to sequentially
isolate individual speakers from the given audio mixture, resulting in notable
missing and noisy parts across various segments of the separated speech. In
this study, we propose a simultaneous multi-speaker separation framework that
can facilitate the concurrent separation of multiple speakers within a singular
process. We introduce speaker-wise interactions to establish distinctions and
correlations among speakers. Experimental results on the VoxCeleb2 and LRS3
datasets demonstrate that our method achieves state-of-the-art performance in
separating mixtures with 2, 3, 4, and 5 speakers, respectively. Additionally,
our model can utilize speakers with complete audio-visual information to
mitigate other visual-deficient speakers, thereby enhancing its resilience to
missing visual cues. We also conduct experiments where visual information for
specific speakers is entirely absent or visual frames are partially missing.
The results demonstrate that our model consistently outperforms others,
exhibiting the smallest performance drop across all settings involving 2, 3, 4,
and 5 speakers.
