  What does a typical visit to Paris look like? Do people first take photos of
the Louvre and then the Eiffel Tower? Can we visually model a temporal event
like "Paris Vacation" using current frameworks? In this paper, we explore how
we can automatically learn the temporal aspects, or storylines of visual
concepts from web data. Previous attempts focus on consecutive image-to-image
transitions and are unsuccessful at recovering the long-term underlying story.
Our novel Skipping Recurrent Neural Network (S-RNN) model does not attempt to
predict each and every data point in the sequence, like classic RNNs. Rather,
S-RNN uses a framework that skips through the images in the photo stream to
explore the space of all ordered subsets of the albums via an efficient
sampling procedure. This approach reduces the negative impact of strong
short-term correlations, and recovers the latent story more accurately. We show
how our learned storylines can be used to analyze, predict, and summarize photo
albums from Flickr. Our experimental results provide strong qualitative and
quantitative evidence that S-RNN is significantly better than other candidate
methods such as LSTMs on learning long-term correlations and recovering latent
storylines. Moreover, we show how storylines can help machines better
understand and summarize photo streams by inferring a brief personalized story
of each individual album.
