  Heterogeneous face recognition is a challenging task due to the large
modality discrepancy and insufficient cross-modal samples. Most existing works
focus on discriminative feature transformation, metric learning and cross-modal
face synthesis. However, the fact that cross-modal faces are always coupled by
domain (modality) and identity information has received little attention.
Therefore, how to learn and utilize the domain-private feature and
domain-agnostic feature for modality adaptive face recognition is the focus of
this work. Specifically, this paper proposes a Feature Aggregation Network
(FAN), which includes disentangled representation module (DRM), feature fusion
module (FFM) and adaptive penalty metric (APM) learning session. First, in DRM,
two subnetworks, i.e. domain-private network and domain-agnostic network are
specially designed for learning modality features and identity features,
respectively. Second, in FFM, the identity features are fused with domain
features to achieve cross-modal bi-directional identity feature transformation,
which, to a large extent, further disentangles the modality information and
identity information. Third, considering that the distribution imbalance
between easy and hard pairs exists in cross-modal datasets, which increases the
risk of model bias, the identity preserving guided metric learning with
adaptive hard pairs penalization is proposed in our FAN. The proposed APM also
guarantees the cross-modality intra-class compactness and inter-class
separation. Extensive experiments on benchmark cross-modal face datasets show
that our FAN outperforms SOTA methods.
