  Controllable Image Captioning (CIC) aims at generating natural language
descriptions for an image, conditioned on information provided by end users,
e.g., regions, entities or events of interest. However, available
image-language datasets mainly contain captions that describe the entirety of
an image, making them ineffective for training CIC models that can potentially
attend to any subset of regions or relationships. To tackle this challenge, we
propose a novel, fully automatic method to sample additional focused and
visually grounded captions using a unified structured semantic representation
built on top of the existing set of captions associated with an image. We
leverage Abstract Meaning Representation (AMR), a cross-lingual graph-based
semantic formalism, to encode all possible spatio-semantic relations between
entities, beyond the typical spatial-relations-only focus of current methods.
We use this Structured Semantic Augmentation (SSA) framework to augment
existing image-caption datasets with the grounded controlled captions,
increasing their spatial and semantic diversity and focal coverage. We then
develop a new model, CIC-BART-SSA, specifically tailored for the CIC task, that
sources its control signals from SSA-diversified datasets. We empirically show
that, compared to SOTA CIC models, CIC-BART-SSA generates captions that are
superior in diversity and text quality, are competitive in controllability,
and, importantly, minimize the gap between broad and highly focused controlled
captioning performance by efficiently generalizing to the challenging highly
focused scenarios. Code is available at
https://github.com/SamsungLabs/CIC-BART-SSA.
