  Scene, as the crucial unit of storytelling in movies, contains complex
activities of actors and their interactions in a physical environment.
Identifying the composition of scenes serves as a critical step towards
semantic understanding of movies. This is very challenging -- compared to the
videos studied in conventional vision problems, e.g. action recognition, as
scenes in movies usually contain much richer temporal structures and more
complex semantic information. Towards this goal, we scale up the scene
segmentation task by building a large-scale video dataset MovieScenes, which
contains 21K annotated scene segments from 150 movies. We further propose a
local-to-global scene segmentation framework, which integrates multi-modal
information across three levels, i.e. clip, segment, and movie. This framework
is able to distill complex semantics from hierarchical temporal structures over
a long movie, providing top-down guidance for scene segmentation. Our
experiments show that the proposed network is able to segment a movie into
scenes with high accuracy, consistently outperforming previous methods. We also
found that pretraining on our MovieScenes can bring significant improvements to
the existing approaches.
