  Face recognition (FR) has seen significant advancements due to the
utilization of large-scale datasets. Training deep FR models on large-scale
datasets with multiple GPUs is now a common practice. In fact, computing power
has evolved into a foundational and indispensable resource in the area of deep
learning. It is nearly impossible to train a deep FR model without holding
adequate hardware resources. Recognizing this challenge, some FR approaches
have started exploring ways to reduce the time complexity of the
fully-connected layer in FR models. Unlike other approaches, this paper
introduces a simple yet highly effective approach, Moving Haar Learning Rate
(MHLR) scheduler, for scheduling the learning rate promptly and accurately in
the training process. MHLR supports large-scale FR training with only one GPU,
which is able to accelerate the model to 1/4 of its original training time
without sacrificing more than 1% accuracy. More specifically, MHLR only needs
$30$ hours to train the model ResNet100 on the dataset WebFace12M containing
more than 12M face images with 0.6M identities. Extensive experiments validate
the efficiency and effectiveness of MHLR.
