  In this paper, we present a multi-modal online person verification system
using both speech and visual signals. Inspired by neuroscientific findings on
the association of voice and face, we propose an attention-based end-to-end
neural network that learns multi-sensory associations for the task of person
verification. The attention mechanism in our proposed network learns to
conditionally select a salient modality between speech and facial
representations that provides a balance between complementary inputs. By virtue
of this capability, the network is robust to missing or corrupted data from
either modality. In the VoxCeleb2 dataset, we show that our method performs
favorably against competing multi-modal methods. Even for extreme cases of
large corruption or an entirely missing modality, our method demonstrates
robustness over other unimodal methods.
