  It is commonly assumed that language refers to high-level visual concepts
while leaving low-level visual processing unaffected. This view dominates the
current literature in computational models for language-vision tasks, where
visual and linguistic input are mostly processed independently before being
fused into a single representation. In this paper, we deviate from this classic
pipeline and propose to modulate the \emph{entire visual processing} by
linguistic input. Specifically, we condition the batch normalization parameters
of a pretrained residual network (ResNet) on a language embedding. This
approach, which we call MOdulated RESnet (\MRN), significantly improves strong
baselines on two visual question answering tasks. Our ablation study shows that
modulating from the early stages of the visual processing is beneficial.
