  As the range of tasks performed by a general vision system expands, executing
multiple tasks accurately and efficiently in a single network has become an
important and still open problem. Recent computer vision approaches address
this problem by branching networks, or by a channel-wise modulation of the
network feature-maps with task specific vectors. We present a novel
architecture that uses a dedicated top-down control network to modify the
activation of all the units in the main recognition network in a manner that
depends on the selected task, image content, and spatial location. We show the
effectiveness of our scheme by achieving significantly better results than
alternative state-of-the-art approaches on four datasets. We further
demonstrate our advantages in terms of task selectivity, scaling the number of
tasks and interpretability.
