  The study of affective computing in the wild setting is underpinned by
databases. Existing multimodal emotion databases in the real-world conditions
are few and small, with a limited number of subjects and expressed in a single
language. To meet this requirement, we collected, annotated, and prepared to
release a new natural state video database (called HEU Emotion). HEU Emotion
contains a total of 19,004 video clips, which is divided into two parts
according to the data source. The first part contains videos downloaded from
Tumblr, Google, and Giphy, including 10 emotions and two modalities (facial
expression and body posture). The second part includes corpus taken manually
from movies, TV series, and variety shows, consisting of 10 emotions and three
modalities (facial expression, body posture, and emotional speech). HEU Emotion
is by far the most extensive multi-modal emotional database with 9,951
subjects. In order to provide a benchmark for emotion recognition, we used many
conventional machine learning and deep learning methods to evaluate HEU
Emotion. We proposed a Multi-modal Attention module to fuse multi-modal
features adaptively. After multi-modal fusion, the recognition accuracies for
the two parts increased by 2.19% and 4.01% respectively over those of
single-modal facial expression recognition.
