  Augmented reality devices have the potential to enhance human perception and
enable other assistive functionalities in complex conversational environments.
Effectively capturing the audio-visual context necessary for understanding
these social interactions first requires detecting and localizing the voice
activities of the device wearer and the surrounding people. These tasks are
challenging due to their egocentric nature: the wearer's head motion may cause
motion blur, surrounding people may appear in difficult viewing angles, and
there may be occlusions, visual clutter, audio noise, and bad lighting. Under
these conditions, previous state-of-the-art active speaker detection methods do
not give satisfactory results. Instead, we tackle the problem from a new
setting using both video and multi-channel microphone array audio. We propose a
novel end-to-end deep learning approach that is able to give robust voice
activity detection and localization results. In contrast to previous methods,
our method localizes active speakers from all possible directions on the
sphere, even outside the camera's field of view, while simultaneously detecting
the device wearer's own voice activity. Our experiments show that the proposed
method gives superior results, can run in real time, and is robust against
noise and clutter.
