  Visual perception is critically influenced by the focus of attention. Due to
limited resources, it is well known that neural representations are biased in
favor of attended locations. Using concurrent eye-tracking and functional
Magnetic Resonance Imaging (fMRI) recordings from a large cohort of human
subjects watching movies, we first demonstrate that leveraging gaze
information, in the form of attentional masking, can significantly improve
brain response prediction accuracy in a neural encoding model. Next, we propose
a novel approach to neural encoding by including a trainable soft-attention
module. Using our new approach, we demonstrate that it is possible to learn
visual attention policies by end-to-end learning merely on fMRI response data,
and without relying on any eye-tracking. Interestingly, we find that attention
locations estimated by the model on independent data agree well with the
corresponding eye fixation patterns, despite no explicit supervision to do so.
Together, these findings suggest that attention modules can be instrumental in
neural encoding models of visual stimuli.
