  Automatically captioning images with natural language sentences is an
important research topic. State of the art models are able to produce
human-like sentences. These models typically describe the depicted scene as a
whole and do not target specific objects of interest or emotional relationships
between these objects in the image. However, marketing companies require to
describe these important attributes of a given scene. In our case, objects of
interest are consumer goods, which are usually identifiable by a product logo
and are associated with certain brands. From a marketing point of view, it is
desirable to also evaluate the emotional context of a trademarked product,
i.e., whether it appears in a positive or a negative connotation. We address
the problem of finding brands in images and deriving corresponding captions by
introducing a modified image captioning network. We also add a third output
modality, which simultaneously produces real-valued image ratings. Our network
is trained using a classification-aware loss function in order to stimulate the
generation of sentences with an emphasis on words identifying the brand of a
product. We evaluate our model on a dataset of images depicting interactions
between humans and branded products. The introduced network improves mean class
accuracy by 24.5 percent. Thanks to adding the third output modality, it also
considerably improves the quality of generated captions for images depicting
branded products.
