  Interaction and collaboration between humans and intelligent machines has
become increasingly important as machine learning methods move into real-world
applications that involve end users. While much prior work lies at the
intersection of natural language and vision, such as image captioning or image
generation from text descriptions, less focus has been placed on the use of
language to guide or improve the performance of a learned visual processing
algorithm. In this paper, we explore methods to flexibly guide a trained
convolutional neural network through user input to improve its performance
during inference. We do so by inserting a layer that acts as a spatio-semantic
guide into the network. This guide is trained to modify the network's
activations, either directly via an energy minimization scheme or indirectly
through a recurrent model that translates human language queries to interaction
weights. Learning the verbal interaction is fully automatic and does not
require manual text annotations. We evaluate the method on two datasets,
showing that guiding a pre-trained network can improve performance, and provide
extensive insights into the interaction between the guide and the CNN.
