  Apparent personality and emotion analysis are both central to affective
computing. Existing works solve them individually. In this paper we investigate
if such high-level affect traits and their relationship can be jointly learned
from face images in the wild. To this end, we introduce PersEmoN, an end-to-end
trainable and deep Siamese-like network. It consists of two convolutional
network branches, one for emotion and the other for apparent personality. Both
networks share their bottom feature extraction module and are optimized within
a multi-task learning framework. Emotion and personality networks are dedicated
to their own annotated dataset. Furthermore, an adversarial-like loss function
is employed to promote representation coherence among heterogeneous dataset
sources. Based on this, we also explore the emotion-to-apparent-personality
relationship. Extensive experiments demonstrate the effectiveness of PersEmoN.
