  The goal of this work is Active Speaker Detection (ASD), a task to determine
whether a person is speaking or not in a series of video frames. Previous works
have dealt with the task by exploring network architectures while learning
effective representations has been less explored. In this work, we propose
TalkNCE, a novel talk-aware contrastive loss. The loss is only applied to part
of the full segments where a person on the screen is actually speaking. This
encourages the model to learn effective representations through the natural
correspondence of speech and facial movements. Our loss can be jointly
optimized with the existing objectives for training ASD models without the need
for additional supervision or training data. The experiments demonstrate that
our loss can be easily integrated into the existing ASD frameworks, improving
their performance. Our method achieves state-of-the-art performances on
AVA-ActiveSpeaker and ASW datasets.
