  Learned dynamic weighting of the conditioning signal (attention) has been
shown to improve neural language generation in a variety of settings. The
weights applied when generating a particular output sequence have also been
viewed as providing a potentially explanatory insight into the internal
workings of the generator. In this paper, we reverse the direction of this
connection and ask whether through the control of the attention of the model we
can control its output. Specifically, we take a standard neural image
captioning model that uses attention, and fix the attention to pre-determined
areas in the image. We evaluate whether the resulting output is more likely to
mention the class of the object in that area than the normally generated
caption. We introduce three effective methods to control the attention and find
that these are producing expected results in up to 28.56% of the cases.
