  2D+3D facial expression recognition (FER) can effectively cope with
illumination changes and pose variations by simultaneously merging 2D texture
and more robust 3D depth information. Most deep learning-based approaches
employ the simple fusion strategy that concatenates the multimodal features
directly after fully-connected layers, without considering the different
degrees of significance for each modality. Meanwhile, how to focus on both 2D
and 3D local features in salient regions is still a great challenge. In this
letter, we propose the adaptive fusion network with masks (AFNet-M) for 2D+3D
FER. To enhance 2D and 3D local features, we take the masks annotating salient
regions of the face as prior knowledge and design the mask attention module
(MA) which can automatically learn two modulation vectors to adjust the feature
maps. Moreover, we introduce a novel fusion strategy that can perform adaptive
fusion at convolutional layers through the designed importance weights
computing module (IWC). Experimental results demonstrate that our AFNet-M
achieves the state-of-the-art performance on BU-3DFE and Bosphorus datasets and
requires fewer parameters in comparison with other models.
