  Image paragraph generation is the task of producing a coherent story (usually
a paragraph) that describes the visual content of an image. The problem
nevertheless is not trivial especially when there are multiple descriptive and
diverse gists to be considered for paragraph generation, which often happens in
real images. A valid question is how to encapsulate such gists/topics that are
worthy of mention from an image, and then describe the image from one topic to
another but holistically with a coherent structure. In this paper, we present a
new design --- Convolutional Auto-Encoding (CAE) that purely employs
convolutional and deconvolutional auto-encoding framework for topic modeling on
the region-level features of an image. Furthermore, we propose an architecture,
namely CAE plus Long Short-Term Memory (dubbed as CAE-LSTM), that novelly
integrates the learnt topics in support of paragraph generation. Technically,
CAE-LSTM capitalizes on a two-level LSTM-based paragraph generation framework
with attention mechanism. The paragraph-level LSTM captures the inter-sentence
dependency in a paragraph, while sentence-level LSTM is to generate one
sentence which is conditioned on each learnt topic. Extensive experiments are
conducted on Stanford image paragraph dataset, and superior results are
reported when comparing to state-of-the-art approaches. More remarkably,
CAE-LSTM increases CIDEr performance from 20.93% to 25.15%.
