  Recent models of emotion recognition strongly rely on supervised deep
learning solutions for the distinction of general emotion expressions. However,
they are not reliable when recognizing online and personalized facial
expressions, e.g., for person-specific affective understanding. In this paper,
we present a neural model based on a conditional adversarial autoencoder to
learn how to represent and edit general emotion expressions. We then propose
Grow-When-Required networks as personalized affective memories to learn
individualized aspects of emotion expressions. Our model achieves
state-of-the-art performance on emotion recognition when evaluated on
\textit{in-the-wild} datasets. Furthermore, our experiments include ablation
studies and neural visualizations in order to explain the behavior of our
model.
