  Deep neural networks have rapidly become the mainstream method for face
recognition (FR). However, this limits the deployment of such models that
contain an extremely large number of parameters to embedded and low-end
devices. In this work, we present an extremely lightweight and accurate FR
solution, namely PocketNet. We utilize neural architecture search to develop a
new family of lightweight face-specific architectures. We additionally propose
a novel training paradigm based on knowledge distillation (KD), the multi-step
KD, where the knowledge is distilled from the teacher model to the student
model at different stages of the training maturity. We conduct a detailed
ablation study proving both, the sanity of using NAS for the specific task of
FR rather than general object classification, and the benefits of our proposed
multi-step KD. We present an extensive experimental evaluation and comparisons
with the state-of-the-art (SOTA) compact FR models on nine different benchmarks
including large-scale evaluation benchmarks such as IJB-B, IJB-C, and MegaFace.
PocketNets have consistently advanced the SOTA FR performance on nine
mainstream benchmarks when considering the same level of model compactness.
With 0.92M parameters, our smallest network PocketNetS-128 achieved very
competitive results to recent SOTA compacted models that contain up to 4M
parameters.
