  Over the years, state-of-the-art (SoTA) image captioning methods have
achieved promising results on some evaluation metrics (e.g., CIDEr). However,
recent findings show that the captions generated by these methods tend to be
biased toward the "average" caption that only captures the most general mode
(a.k.a, language pattern) in the training corpus, i.e., the so-called mode
collapse problem. Affected by it, the generated captions are limited in
diversity and usually less informative than natural image descriptions made by
humans. In this paper, we seek to avoid this problem by proposing a Discrete
Mode Learning (DML) paradigm for image captioning. Our innovative idea is to
explore the rich modes in the training caption corpus to learn a set of "mode
embeddings", and further use them to control the mode of the generated captions
for existing image captioning models. Specifically, the proposed DML optimizes
a dual architecture that consists of an image-conditioned discrete variational
autoencoder (CdVAE) branch and a mode-conditioned image captioning (MIC)
branch. The CdVAE branch maps each image caption to one of the mode embeddings
stored in a learned codebook, and is trained with a pure non-autoregressive
generation objective to make the modes distinct and representative. The MIC
branch can be simply modified from an existing image captioning model, where
the mode embedding is added to the original word embeddings as the control
signal. In the experiments, we apply the proposed DML to two widely used image
captioning models, Transformer and AoANet. The results show that the learned
mode embedding successfully facilitates these models to generate high-quality
image captions with different modes, further leading to better performance for
both diversity and quality on the MSCOCO dataset.
