  Visual storytelling is a task of creating a short story based on photo
streams. Unlike existing visual captioning, storytelling aims to contain not
only factual descriptions, but also human-like narration and semantics.
However, the VIST dataset consists only of a small, fixed number of photos per
story. Therefore, the main challenge of visual storytelling is to fill in the
visual gap between photos with narrative and imaginative story. In this paper,
we propose to explicitly learn to imagine a storyline that bridges the visual
gap. During training, one or more photos is randomly omitted from the input
stack, and we train the network to produce a full plausible story even with
missing photo(s). Furthermore, we propose for visual storytelling a
hide-and-tell model, which is designed to learn non-local relations across the
photo streams and to refine and improve conventional RNN-based models. In
experiments, we show that our scheme of hide-and-tell, and the network design
are indeed effective at storytelling, and that our model outperforms previous
state-of-the-art methods in automatic metrics. Finally, we qualitatively show
the learned ability to interpolate storyline over visual gaps.
