  There is growing interest in artificial intelligence to build socially
intelligent robots. This requires machines to have the ability to "read"
people's emotions, motivations, and other factors that affect behavior. Towards
this goal, we introduce a novel dataset called MovieGraphs which provides
detailed, graph-based annotations of social situations depicted in movie clips.
Each graph consists of several types of nodes, to capture who is present in the
clip, their emotional and physical attributes, their relationships (i.e.,
parent/child), and the interactions between them. Most interactions are
associated with topics that provide additional details, and reasons that give
motivations for actions. In addition, most interactions and many attributes are
grounded in the video with time stamps. We provide a thorough analysis of our
dataset, showing interesting common-sense correlations between different social
aspects of scenes, as well as across scenes over time. We propose a method for
querying videos and text with graphs, and show that: 1) our graphs contain rich
and sufficient information to summarize and localize each scene; and 2)
subgraphs allow us to describe situations at an abstract level and retrieve
multiple semantically relevant situations. We also propose methods for
interaction understanding via ordering, and reason understanding. MovieGraphs
is the first benchmark to focus on inferred properties of human-centric
situations, and opens up an exciting avenue towards socially-intelligent AI
agents.
