  Although state-of-the-art classifiers for facial expression recognition (FER)
can achieve a high level of accuracy, they lack interpretability, an important
feature for end-users. Experts typically associate spatial action units (\aus)
from a codebook to facial regions for the visual interpretation of expressions.
In this paper, the same expert steps are followed. A new learning strategy is
proposed to explicitly incorporate \au cues into classifier training, allowing
to train deep interpretable models. During training, this \au codebook is used,
along with the input image expression label, and facial landmarks, to construct
a \au heatmap that indicates the most discriminative image regions of interest
w.r.t the facial expression. This valuable spatial cue is leveraged to train a
deep interpretable classifier for FER. This is achieved by constraining the
spatial layer features of a classifier to be correlated with \au heatmaps.
Using a composite loss, the classifier is trained to correctly classify an
image while yielding interpretable visual layer-wise attention correlated with
\au maps, simulating the expert decision process. Our strategy only relies on
image class expression for supervision, without additional manual annotations.
Our new strategy is generic, and can be applied to any deep CNN- or
transformer-based classifier without requiring any architectural change or
significant additional training time. Our extensive evaluation on two public
benchmarks \rafdb, and \affectnet datasets shows that our proposed strategy can
improve layer-wise interpretability without degrading classification
performance. In addition, we explore a common type of interpretable classifiers
that rely on class activation mapping (CAM) methods, and show that our approach
can also improve CAM interpretability.
