  An objective understanding of media depictions, such as inclusive portrayals
of how much someone is heard and seen on screen such as in film and television,
requires the machines to discern automatically who, when, how, and where
someone is talking, and not. Speaker activity can be automatically discerned
from the rich multimodal information present in the media content. This is
however a challenging problem due to the vast variety and contextual
variability in the media content, and the lack of labeled data. In this work,
we present a cross-modal neural network for learning visual representations,
which have implicit information pertaining to the spatial location of a speaker
in the visual frames. Avoiding the need for manual annotations for active
speakers in visual frames, acquiring of which is very expensive, we present a
weakly supervised system for the task of localizing active speakers in movie
content. We use the learned cross-modal visual representations, and provide
weak supervision from movie subtitles acting as a proxy for voice activity,
thus requiring no manual annotations. We evaluate the performance of the
proposed system on the AVA active speaker dataset and demonstrate the
effectiveness of the cross-modal embeddings for localizing active speakers in
comparison to fully supervised systems. We also demonstrate state-of-the-art
performance for the task of voice activity detection in an audio-visual
framework, especially when speech is accompanied by noise and music.
