  With the strong robusticity on illumination variations, near-infrared (NIR)
can be an effective and essential complement to visible (VIS) facial expression
recognition in low lighting or complete darkness conditions. However, facial
expression recognition (FER) from NIR images presents more challenging problem
than traditional FER due to the limitations imposed by the data scale and the
difficulty of extracting discriminative features from incomplete visible
lighting contents. In this paper, we give the first attempt to deep NIR facial
expression recognition and proposed a novel method called near-infrared facial
expression transformer (NFER-Former). Specifically, to make full use of the
abundant label information in the field of VIS, we introduce a Self-Attention
Orthogonal Decomposition mechanism that disentangles the expression information
and spectrum information from the input image, so that the expression features
can be extracted without the interference of spectrum variation. We also
propose a Hypergraph-Guided Feature Embedding method that models some key
facial behaviors and learns the structure of the complex correlations between
them, thereby alleviating the interference of inter-class similarity.
Additionally, we have constructed a large NIR-VIS Facial Expression dataset
that includes 360 subjects to better validate the efficiency of NFER-Former.
Extensive experiments and ablation studies show that NFER-Former significantly
improves the performance of NIR FER and achieves state-of-the-art results on
the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.
