  Social relations are the foundation of human daily life. Developing
techniques to analyze such relations from visual data bears great potential to
build machines that better understand us and are capable of interacting with us
at a social level. Previous investigations have remained partial due to the
overwhelming diversity and complexity of the topic and consequently have only
focused on a handful of social relations. In this paper, we argue that the
domain-based theory from social psychology is a great starting point to
systematically approach this problem. The theory provides coverage of all
aspects of social relations and equally is concrete and predictive about the
visual attributes and behaviors defining the relations included in each domain.
We provide the first dataset built on this holistic conceptualization of social
life that is composed of a hierarchical label space of social domains and
social relations. We also contribute the first models to recognize such domains
and relations and find superior performance for attribute based features.
Beyond the encouraging performance of the attribute based approach, we also
find interpretable features that are in accordance with the predictions from
social psychology literature. Beyond our findings, we believe that our
contributions more tightly interleave visual recognition and social psychology
theory that has the potential to complement the theoretical work in the area
with empirical and data-driven models of social life.
