  Modern face recognition systems utilize deep neural networks to extract
salient features from a face. These features denote embeddings in latent space
and are often stored as templates in a face recognition system. These
embeddings are susceptible to data leakage and, in some cases, can even be used
to reconstruct the original face image. To prevent compromising identities,
template protection schemes are commonly employed. However, these schemes may
still not prevent the leakage of soft biometric information such as age, gender
and race. To alleviate this issue, we propose a novel technique that combines
Fully Homomorphic Encryption (FHE) with an existing template protection scheme
known as PolyProtect. We show that the embeddings can be compressed and
encrypted using FHE and transformed into a secure PolyProtect template using
polynomial transformation, for additional protection. We demonstrate the
efficacy of the proposed approach through extensive experiments on multiple
datasets. Our proposed approach ensures irreversibility and unlinkability,
effectively preventing the leakage of soft biometric attributes from face
embeddings without compromising recognition accuracy.
