  We introduce multimodal story summarization by leveraging TV episode recaps -
short video sequences interweaving key story moments from previous episodes to
bring viewers up to speed. We propose PlotSnap, a dataset featuring two crime
thriller TV shows with rich recaps and long episodes of 40 minutes. Story
summarization labels are unlocked by matching recap shots to corresponding
sub-stories in the episode. We propose a hierarchical model TaleSumm that
processes entire episodes by creating compact shot and dialog representations,
and predicts importance scores for each video shot and dialog utterance by
enabling interactions between local story groups. Unlike traditional
summarization, our method extracts multiple plot points from long videos. We
present a thorough evaluation on story summarization, including promising
cross-series generalization. TaleSumm also shows good results on classic video
summarization benchmarks.
