  Facial expression recognition is a challenging task when neural network is
applied to pattern recognition. Most of the current recognition research is
based on single source facial data, which generally has the disadvantages of
low accuracy and low robustness. In this paper, a neural network algorithm of
facial expression recognition based on multimodal data fusion is proposed. The
algorithm is based on the multimodal data, and it takes the facial image, the
histogram of oriented gradient of the image and the facial landmarks as the
input, and establishes CNN, LNN and HNN three sub neural networks to extract
data features, using multimodal data feature fusion mechanism to improve the
accuracy of facial expression recognition. Experimental results show that,
benefiting by the complementarity of multimodal data, the algorithm has a great
improvement in accuracy, robustness and detection speed compared with the
traditional facial expression recognition algorithm. Especially in the case of
partial occlusion, illumination and head posture transformation, the algorithm
also shows a high confidence.
