  Existing research for image captioning usually represents an image using a
scene graph with low-level facts (objects and relations) and fails to capture
the high-level semantics. In this paper, we propose a Theme Concepts extended
Image Captioning (TCIC) framework that incorporates theme concepts to represent
high-level cross-modality semantics. In practice, we model theme concepts as
memory vectors and propose Transformer with Theme Nodes (TTN) to incorporate
those vectors for image captioning. Considering that theme concepts can be
learned from both images and captions, we propose two settings for their
representations learning based on TTN. On the vision side, TTN is configured to
take both scene graph based features and theme concepts as input for visual
representation learning. On the language side, TTN is configured to take both
captions and theme concepts as input for text representation re-construction.
Both settings aim to generate target captions with the same transformer-based
decoder. During the training, we further align representations of theme
concepts learned from images and corresponding captions to enforce the
cross-modality learning. Experimental results on MS COCO show the effectiveness
of our approach compared to some state-of-the-art models.
