  Image captioning is an ambiguous problem, with many suitable captions for an
image. To address ambiguity, beam search is the de facto method for sampling
multiple captions. However, beam search is computationally expensive and known
to produce generic captions. To address this concern, some variational
auto-encoder (VAE) and generative adversarial net (GAN) based methods have been
proposed. Though diverse, GAN and VAE are less accurate. In this paper, we
first predict a meaningful summary of the image, then generate the caption
based on that summary. We use part-of-speech as summaries, since our summary
should drive caption generation. We achieve the trifecta: (1) High accuracy for
the diverse captions as evaluated by standard captioning metrics and user
studies; (2) Faster computation of diverse captions compared to beam search and
diverse beam search; and (3) High diversity as evaluated by counting novel
sentences, distinct n-grams and mutual overlap (i.e., mBleu-4) scores.
