  Socially-intelligent agents are of growing interest in artificial
intelligence. To this end, we need systems that can understand social
relationships in diverse social contexts. Inferring the social context in a
given visual scene not only involves recognizing objects, but also demands a
more in-depth understanding of the relationships and attributes of the people
involved. To achieve this, one computational approach for representing human
relationships and attributes is to use an explicit knowledge graph, which
allows for high-level reasoning. We introduce a novel end-to-end-trainable
neural network that is capable of generating a Social Relationship Graph - a
structured, unified representation of social relationships and attributes -
from a given input image. Our Social Relationship Graph Generation Network
(SRG-GN) is the first to use memory cells like Gated Recurrent Units (GRUs) to
iteratively update the social relationship states in a graph using scene and
attribute context. The neural network exploits the recurrent connections among
the GRUs to implement message passing between nodes and edges in the graph, and
results in significant improvement over previous methods for social
relationship recognition.
