  We address the problem of story-based temporal summarization of long
360{\deg} videos. We propose a novel memory network model named Past-Future
Memory Network (PFMN), in which we first compute the scores of 81 normal field
of view (NFOV) region proposals cropped from the input 360{\deg} video, and
then recover a latent, collective summary using the network with two external
memories that store the embeddings of previously selected subshots and future
candidate subshots. Our major contributions are two-fold. First, our work is
the first to address story-based temporal summarization of 360{\deg} videos.
Second, our model is the first attempt to leverage memory networks for video
summarization tasks. For evaluation, we perform three sets of experiments.
First, we investigate the view selection capability of our model on the
Pano2Vid dataset. Second, we evaluate the temporal summarization with a newly
collected 360{\deg} video dataset. Finally, we experiment our model's
performance in another domain, with image-based storytelling VIST dataset. We
verify that our model achieves state-of-the-art performance on all the tasks.
