  We propose the ViNet architecture for audio-visual saliency prediction. ViNet
is a fully convolutional encoder-decoder architecture. The encoder uses visual
features from a network trained for action recognition, and the decoder infers
a saliency map via trilinear interpolation and 3D convolutions, combining
features from multiple hierarchies. The overall architecture of ViNet is
conceptually simple; it is causal and runs in real-time (60 fps). ViNet does
not use audio as input and still outperforms the state-of-the-art audio-visual
saliency prediction models on nine different datasets (three visual-only and
six audio-visual datasets). ViNet also surpasses human performance on the CC,
SIM and AUC metrics for the AVE dataset, and to our knowledge, it is the first
network to do so. We also explore a variation of ViNet architecture by
augmenting audio features into the decoder. To our surprise, upon sufficient
training, the network becomes agnostic to the input audio and provides the same
output irrespective of the input. Interestingly, we also observe similar
behaviour in the previous state-of-the-art models \cite{tsiami2020stavis} for
audio-visual saliency prediction. Our findings contrast with previous works on
deep learning-based audio-visual saliency prediction, suggesting a clear avenue
for future explorations incorporating audio in a more effective manner. The
code and pre-trained models are available at
https://github.com/samyak0210/ViNet.
