  In this work, we study the robustness of a CNN+RNN based image captioning
system being subjected to adversarial noises. We propose to fool an image
captioning system to generate some targeted partial captions for an image
polluted by adversarial noises, even the targeted captions are totally
irrelevant to the image content. A partial caption indicates that the words at
some locations in this caption are observed, while words at other locations are
not restricted.It is the first work to study exact adversarial attacks of
targeted partial captions. Due to the sequential dependencies among words in a
caption, we formulate the generation of adversarial noises for targeted partial
captions as a structured output learning problem with latent variables. Both
the generalized expectation maximization algorithm and structural SVMs with
latent variables are then adopted to optimize the problem. The proposed methods
generate very successful at-tacks to three popular CNN+RNN based image
captioning models. Furthermore, the proposed attack methods are used to
understand the inner mechanism of image captioning systems, providing the
guidance to further improve automatic image captioning systems towards human
captioning.
