  Micro-expression (ME) recognition plays a crucial role in a wide range of
applications, particularly in public security and psychotherapy. Recently,
traditional methods rely excessively on machine learning design and the
recognition rate is not high enough for its practical application because of
its short duration and low intensity. On the other hand, some methods based on
deep learning also cannot get high accuracy due to problems such as the
imbalance of databases. To address these problems, we design a multi-stream
convolutional neural network (MSCNN) for ME recognition in this paper.
Specifically, we employ EVM and optical flow to magnify and visualize subtle
movement changes in MEs and extract the masks from the optical flow images. And
then, we add the masks, optical flow images, and grayscale images into the
MSCNN. After that, in order to overcome the imbalance of databases, we added a
random over-sampler after the Dense Layer of the neural network. Finally,
extensive experiments are conducted on two public ME databases: CASME II and
SAMM. Compared with many recent state-of-the-art approaches, our method
achieves more promising recognition results.
