  Interpersonal relation defines the association, e.g., warm, friendliness, and
dominance, between two or more people. Motivated by psychological studies, we
investigate if such fine-grained and high-level relation traits can be
characterized and quantified from face images in the wild. We address this
challenging problem by first studying a deep network architecture for robust
recognition of facial expressions. Unlike existing models that typically learn
from facial expression labels alone, we devise an effective multitask network
that is capable of learning from rich auxiliary attributes such as gender, age,
and head pose, beyond just facial expression data. While conventional
supervised training requires datasets with complete labels (e.g., all samples
must be labeled with gender, age, and expression), we show that this
requirement can be relaxed via a novel attribute propagation method. The
approach further allows us to leverage the inherent correspondences between
heterogeneous attribute sources despite the disparate distributions of
different datasets. With the network we demonstrate state-of-the-art results on
existing facial expression recognition benchmarks. To predict inter-personal
relation, we use the expression recognition network as branches for a Siamese
model. Extensive experiments show that our model is capable of mining mutual
context of faces for accurate fine-grained interpersonal prediction.
