  Image captioning models generally lack the capability to take into account
user interest, and usually default to global descriptions that try to balance
readability, informativeness, and information overload. On the other hand, VQA
models generally lack the ability to provide long descriptive answers, while
expecting the textual question to be quite precise. We present a method to
control the concepts that an image caption should focus on, using an additional
input called the guiding text that refers to either groundable or ungroundable
concepts in the image. Our model consists of a Transformer-based multimodal
encoder that uses the guiding text together with global and object-level image
features to derive early-fusion representations used to generate the guided
caption. While models trained on Visual Genome data have an in-domain advantage
of fitting well when guided with automatic object labels, we find that guided
captioning models trained on Conceptual Captions generalize better on
out-of-domain images and guiding texts. Our human-evaluation results indicate
that attempting in-the-wild guided image captioning requires access to large,
unrestricted-domain training datasets, and that increased style diversity (even
without increasing the number of unique tokens) is a key factor for improved
performance.
