  Convolutional neural networks (CNNs) have been shown to be the
state-of-the-art approach for modeling the transfer functions of visual
cortical neurons. Cortical neurons in the primary visual cortex are are
sensitive to contextual information mediated by extensive horizontal and
feedback connections. Standard CNNs can integrate global spatial image
information to model such contextual modulation via two mechanisms: successive
rounds of convolutions and a fully connected readout layer. In this paper, we
find that non-local networks or self-attention (SA) mechanisms, theoretically
related to context-dependent flexible gating mechanisms observed in the primary
visual cortex, improve neural response predictions over parameter-matched CNNs
in two key metrics: tuning curve correlation and tuning peak. We factorize
networks to determine the relative contribution of each context mechanism. This
reveals that information in the local receptive field is most important for
modeling the overall tuning curve, but surround information is critically
necessary for characterizing the tuning peak. We find that self-attention can
replace subsequent spatial-integration convolutions when learned in an
incremental manner, and is further enhanced in the presence of a fully
connected readout layer, suggesting that the two context mechanisms are
complementary. Finally, we find that learning a receptive-field-centric model
with self-attention, before incrementally learning a fully connected readout,
yields a more biologically realistic model in terms of center-surround
contributions.
