  Capsule neural network is a new and popular technique in deep learning.
However, the traditional capsule neural network does not extract features
sufficiently before the dynamic routing between the capsules. In this paper,
the one Double Enhanced Capsule Neural Network (E2-Capsnet) that uses AU-aware
attention for facial expression recognition (FER) is proposed. The E2-Capsnet
takes advantage of dynamic routing between the capsules, and has two
enhancement modules which are beneficial for FER. The first enhancement module
is the convolutional neural network with AU-aware attention, which can help
focus on the active areas of the expression. The second enhancement module is
the capsule neural network with multiple convolutional layers, which enhances
the ability of the feature representation. Finally, squashing function is used
to classify the facial expression. We demonstrate the effectiveness of
E2-Capsnet on the two public benchmark datasets, RAF-DB and EmotioNet. The
experimental results show that our E2-Capsnet is superior to the
state-of-the-art methods. Our implementation will be publicly available online.
