  Plastic surgery and disguise variations are two of the most challenging
co-variates of face recognition. The state-of-art deep learning models are not
sufficiently successful due to the availability of limited training samples. In
this paper, a novel framework is proposed which transfers fundamental visual
features learnt from a generic image dataset to supplement a supervised face
recognition model. The proposed algorithm combines off-the-shelf supervised
classifier and a generic, task independent network which encodes information
related to basic visual cues such as color, shape, and texture. Experiments are
performed on IIITD plastic surgery face dataset and Disguised Faces in the Wild
(DFW) dataset. Results showcase that the proposed algorithm achieves state of
the art results on both the datasets. Specifically on the DFW database, the
proposed algorithm yields over 87% verification accuracy at 1% false accept
rate which is 53.8% better than baseline results computed using VGGFace.
