  Face recognition under ideal conditions is now considered a well-solved
problem with advances in deep learning. Recognizing faces under occlusion,
however, still remains a challenge. Existing techniques often fail to recognize
faces with both the mouth and nose covered by a mask, which is now very common
under the COVID-19 pandemic. Common approaches to tackle this problem include
1) discarding information from the masked regions during recognition and 2)
restoring the masked regions before recognition. Very few works considered the
consistency between features extracted from masked faces and from their
mask-free counterparts. This resulted in models trained for recognizing masked
faces often showing degraded performance on mask-free faces. In this paper, we
propose a unified framework, named Face Feature Rectification Network
(FFR-Net), for recognizing both masked and mask-free faces alike. We introduce
rectification blocks to rectify features extracted by a state-of-the-art
recognition model, in both spatial and channel dimensions, to minimize the
distance between a masked face and its mask-free counterpart in the rectified
feature space. Experiments show that our unified framework can learn a
rectified feature space for recognizing both masked and mask-free faces
effectively, achieving state-of-the-art results. Project code:
https://github.com/haoosz/FFR-Net
