  Face recognition technology has demonstrated tremendous progress over the
past few years, primarily due to advances in representation learning. As we
witness the widespread adoption of these systems, it is imperative to consider
the security of face representations. In this paper, we explore the
practicality of using a fully homomorphic encryption based framework to secure
a database of face templates. This framework is designed to preserve the
privacy of users and prevent information leakage from the templates, while
maintaining their utility through template matching directly in the encrypted
domain. Additionally, we also explore a batching and dimensionality reduction
scheme to trade-off face matching accuracy and computational complexity.
Experiments on benchmark face datasets (LFW, IJB-A, IJB-B, CASIA) indicate that
secure face matching can be practically feasible (16 KB template size and 0.01
sec per match pair for 512-dimensional features from SphereFace) while
exhibiting minimal loss in matching performance.
