\section{Introduction}
Landmark localization is the task of finding the positions of predefined key points in an image.
One of its main applications is Human Pose Estimation (HPE), a major research topic for computer vision with applications such as action recognition, people tracking and sports analytics.
Landmark localization is a challenging problem due to variations in pose, illumination and occlusion \citep{Dibeklioglu2008}.
Early solutions based on the pictorial structure model \citep{PictorialStructures} define objects as a set of landmarks connected in a deformable structure. 
These methods combine an appearance model to localize the landmarks with a predefined structural model of the object. 
This approach produces landmarks that satisfy both appearance features and spatial configuration of the object \citep{PartBasedModel,better_pictorial,articulated_human}.


Deep learning algorithms have dramatically improved the performance of HPE.
\cite{6909610} proposed one of the first methods, where the HPE is formulated as a regression problem over the normalized joint coordinates.
Although predicting joint coordinates directly from images is very difficult, numerous improvements have been discovered.
For example, \cite{comp_HPR} combined a regression approach with prior information about the human body structure to achieve more robust results. 
\cite{Thompson2014} approached HPE as a heatmap regression problem, where feature maps are upscaled to generate spatial probability maps, with pixel values corresponding to the probability of the landmark residing at a particular position.
These heatmaps are created by convolving the location of each joint with a 2D Gaussian \citep{DL_pose, Thompson2014}.
This novel approach outperformed the existing state of the art methods \citep{Dantone2013, 6909610} on both FLIC and LSP datasets. 
\cite{Wei2016} also presented a heatmap regression network consisting of different stages.
Each stage receives the original image and the heatmap from the previous stage.
This approach outperforms \cite{Thompson2014} on the FLIC, LSP and MPII \citep{MPII} datasets.
A heatmap based stacked hourglass network comprised of stacked modules was proposed by \cite{Newell2016}, each with pooling layers followed by upsampling layers.
They achieve between 0.4 -- 4.6 per cent improvement on different joints in comparison to  \cite{Wei2016}.
\cite{Xiao2018SimpleBF} use a ResNet \citep{resnet2016} backbone followed by deconvolutional layers to generate a set of heatmaps representing the probability map of each joint. 
Whereas, \cite{sun2019deep} combine multiple sub-nets and capture information from different scales, improving the mAP on COCO by 3.3 mAP compared to \cite{Xiao2018SimpleBF}.
Recent studies focus on the target heatmaps and the accuracy of the coordinates retrieved from them. 
\cite{Zhang_2020_CVPR} extract joint coordinates from their respective heatmap using sub-pixel Gaussian fitting, resulting in a performance increase for different state of the art models on both COCO \citep{COCO} and MPII datasets. 
Another direction of amelioration, presented in \cite{huang2020aid}, consists of exploring the effect of different training schedules on various HPE methods.
The authors found schedules that use information dropping augmentations (e.g. random erasing) after a certain number of epochs yield the best performance.
They report a 0.6 AP increase on the COCO dataset when they apply their suggested training schedule to the state of the art HRNet-W48.


Facial landmark localization focuses on predicting pre-defined face key points (e.g. contours of the face, eyes, nose, mouth, eyebrows, etc.) and is used for a wide variety of face-related tasks.
Face recognition uses these landmarks for alignment, this has been an intrinstic part of these systems for over a decade
\citep{kumar2009,wolf2009,parkhi2015,schroff2015,LiuWeiyang2017,wang2018,deng2018,an2020}.
They are also used in face reconstruction where 3D annotations are not available \citep[e.g.][]{Roth2015,Dou2017,Feng2018}, as well as, face animation \citep[e.g.][]{cao2013}, emotion classification \citep{day2016,munasinghe22018}, detecting synthesized faces \citep{yang2019} and facial action unit detection \citep{hinduja2020}.


Early face landmark localization focused on using statistical models or component detectors.
One statistical method is the Active Shape Model (ASM), proposed by \cite{Cootes1995}.
This method focuses on generating a mean face shape model to determine the best position for the facial landmarks. 
ASM computes the initial face shape using the shape model, then fine-tunes the shape parameters for the image by examining the region around each landmark.
Similarly to ASM, \cite{Cootes1998} focus on generating a global face shape model; while also attempting to incorporate texture information using an appearance model.
The appearance and shape parameters get fitted by reducing the difference between the image and its synthetically generated counterpart. 
Some researches also explored training patch-based detectors or component detectors to predict each landmark on local patches or anatomical components on the face image, respectively \citep{Liang2008, Zhu2012, Amberg2011, Belhumeur2013, Efraty2011}. 
These approaches require constraints on the face shape to obtain the best landmark configuration due to the lack of global contextual information of the face.


Recently, Convolutional Neural Networks (CNNs) have supplanted classical approaches due to their ability to extract contextual information.
Generally speaking, there are two widely adopted approaches; coordinate regression and heatmap fitting---just like HPE \citep[e.g.][]{Thompson2014,DL_pose,Xiao2018SimpleBF,Zhang_2020_CVPR}.

For coordinate regression, a dense layer is added at the end of the CNN to predict the coordinates of each landmark. 
Among the first of these approaches, \cite{Sun2013} fit a three-level cascaded CNN to localize five facial landmarks: the corners of the eyes, the tip of the nose and corners of the mouth.
The first level estimates the coordinates then the subsequent levels fine-tune them. 
While outperforming classical methods, the cascaded CNN architecture is complicated.
The first level contains three CNNs, each predicting a different subset of landmark, and the following two levels each have ten CNNs.
\cite{Zhou2013} proposed a four-level cascaded coarse-to-fine CNN, where the first level estimates two bounding boxes for each face, while the following layers predict the landmarks for the bounding boxes with refinement on the inner parts of the face (e.g., eyes, nose, and mouth).
Both sets of landmarks are then combined to obtain the final positions. 
This method works well in challenging conditions, such as high pose variance, poor illumination, and occlusion.
The authors achieve a significant performance improvement on the 300W challenge \citep{Sagonas2016,Sagonas2013} compared to the baseline.
Similarly, \cite{Zhang2014} proposed a coarse-to-fine cascade of stacked auto-encoders, where the first level performs initial landmarks estimation on lower resolution face image and subsequent layers refine these using higher-resolution local patches for each landmark. 
Their approach outperforms both \cite{Sun2013} and \cite{xiong2013}, while being significantly faster.
More recently, multi-task learning has been utilized to reduce architectural complexity.
By combining: the landmarks with the pose, facial expression, gender, and other attributes, \cite{ZhangLuo2016} built a lighter CNN, which is robust to pose and occlusion. 
Similarly, \cite{Ranjan2016} proposed using multi-task learning on related tasks such as face detection, landmarks localization, pose estimation, and gender classification to achieve state-of-the-art performance on individual tasks, including a 0.42 per cent reduction in error for facial landmarks localization.

Most regression-based approaches suffer from spatial information loss due to the compression of feature maps before the fully-connected layers.
This shortcoming has inspired researchers to leverage the encoder-decoder architecture and propose heatmap based approaches.
\cite{Kowalski2017} proposed using landmark heatmaps for their alignment network to transfer information between different network stages.
Instead of using local patches leading to local minima, their system works on entire images, handling large pose variations and achieving a 72 per cent reduction in the failure rate on the 300W dataset.
Furthermore, \cite{Mahpod2018} proposed a cascaded CNN architecture comprising two CNNs followed by two cascaded subnetworks: a heatmap based network and a regression-based network to refine the heatmap localizations.
Although these methods have led to performance increases, they rely on generating large heatmaps; which can suffer from increased post-processing complexity. 
The ground-truth heatmaps are often dominated by background pixels with tiny positive-valued regions at the landmarks, leading to slower convergence during training.
\cite{xiong2020} proposed using a quasi-Gaussian distribution to represent ground-truth landmark positions as vectors, addressing the foreground-background imbalance.
They also convert the output heatmaps into vectors, which incorporates spatial information and reduces sensitivity.
The authors demonstrate that their approach leads to better convergence, reduced post-processing complexity, and achieves state-of-the-art performance on multiple evaluation datasets, including ranking second on the JD-landmark challenge \citep{liu2019}.


We organize the paper as follows. 
Section 2 presents some of the most recent work that we will draw upon.
Section 3 outlines our training and evaluation methods and our baseline parameters.
Section 4 compares the results using different techniques to increase and reduce both landmark accuracy and computational complexity, respectively.
Section 5 presents our final models, the additional tricks we employed and investigates the potential impact on facial recognition performance.
Section 5 presents our conclusions.


\section{Related Work}


\subsection{Heatmap fitting}
As discussed in the previous section, coordinate regression and heatmap based approaches have been used widely for landmark localization. 
However, regression-based approaches \citep{Sun2013, Zhang2014} tend to lead to spatial information loss.
This reason, combined with the recent success of \cite{Kowalski2017} and \cite{xiong2020}, leads us to employ the heatmap fitting approach. 


\subsection{Sub-pixel inference}
The last step of landmark localization consists of extracting a set of coordinates from the estimated heatmaps.
The simplest approach is to use the argmax of the heatmaps: 
\begin{equation}
\label{equation:1}
    (x, y)_{i} = \underset{m<w, n<h}{\text{argmax}}(H_{i}(m, n)),
\end{equation}
where $H_{i}(m, n)$ is the estimated value of the $i^{th}$ landmark heatmap in ($m$, $n$) and $w$, $h$ are the width and height of $H_{i}$, respectively.
This approach does not allow for sub-pixel localization, as the accuracy is limited by the resolution of the heatmap. 
A simple way to reduce this effect is to modify Equation \ref{equation:1} using the gradient of the heatmap \citep{Xiao2018SimpleBF},
\begin{equation}
\label{equation:2}
    (x,y)_{i} = \underset{m<w, n<h}{\text{argmax}}(H_{i}(m,n)) + c*\frac{\partial H}{\partial x\partial y},
\end{equation}
where $c$ is a correction coefficient applied to the gradient of the heatmap around $(x, y)$ from Equation \ref{equation:1}. 
This approach incorporates the region around the heatmap peak, shifting the position by a sub-pixel distance using the gradient. 


\cite{zhang2019distributionaware} use the prior Gaussian information of the heatmaps to increase the accuracy of the predicted joint coordinates.
As the target heatmaps are generated by a Gaussian convolution, the model is incentivized to also produce Gaussian probability maps.
These estimated heatmaps can be approximated by,
\begin{equation}
\label{equation:3}
    G(X,\mu,\Sigma) = \frac{\exp\big(-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)\big)}{(2\pi)|{\Sigma}^{\frac{1}{2}}|},
\end{equation}
where $X$ is the location and $\mu$ is the ground truth coordinate of the landmark.
The $(x, y)_{i}$ is computed for each landmark to minimize the error between the Equation \ref{equation:3} applied to $(x, y)_{i}$ and the predicted heatmap $H_{i}$. 
This approach results in sub-pixel localization and hence increases the accuracy of the landmarks.
Due to this performance increase, we will employ this method for our post-processing.


\subsection{Pixel shuffle}
Single image super-resolution (SISR) focuses on generating an image with $r$ times higher resolution.
Previous approaches use interpolation (e.g., bicubic interpolation) to perform upsampling either at the first layer of the network \citep{Dong2016, Chen2017, Wang2015} or gradually through the network \citep{Osendorfer2014}; increasing the computational complexity. 
\cite{Shi2016} propose efficient sub-pixel convolutional layers that learn an effective upsampling filter instead of using a pre-defined interpolation. 
These layers contain convolutions and pixel rearrangement and are known as pixel shuffle layers.
SISR can be performed by passing the original image through a three-layer convolutional network, where the output feature maps have the same width and height as the input image. 
Afterwards, the $r^2$ feature maps are rearranged to generate a feature map of the desired size.
Their proposed approach outperforms \cite{Dong2016} on all and \cite{Chen2017} on most benchmark datasets while running at least ten times faster on these datasets for SISR. 
We leverage the pixel-shuffle layers to build an efficient model architecture; due to the reduction in inference time and computational complexity.



\section{Methodology}


\subsection{Training dataset}

Earlier face landmark datasets, including MULTI-PIE \citep{Gross2008}, 300W \citep{Sagonas2016,Sagonas2013}, LFW \citep{HuangGary2008} and the Menpo benchmark \citep{DengRoussos2019}, mainly focus on 68 landmarks.
These landmarks lack some of the core features of the face.
For example, the lower eyebrow and border/wings of the nose are all ignored.
To correct this \cite{liu2019} presented the 106-point JD-landmark dataset.
The dataset consists of 11,393, 2,000, and 2,000 images for the training, validation and test sets, with large variations in the pose.
The updated training dataset was released as part of the 2nd Grand Challenge of 106-Point Facial Landmark Localization (JD-landmark-2)\footnote{\href{https://fllc-icpr2020.github.io}{https://fllc-icpr2020.github.io}\label{footnote:jd2}}.

We preprocess the training and validation images using the publicly available ResNet50 face detector from \cite{deng2019}.
From this model, we get the bounding box and the five na\"{i}ve landmarks for the eyes, nose and mouth of the centre most face.
We calculate the affine transformation between these five landmarks and a set of reference landmarks and apply this transform to the image and the ground truth landmarks.
We choose the reference landmarks such that the face is in the centre of the image with the eyes and mouth horizontal.
Figure \ref{fig:example_images} compares the example images from the validation set which are presented in \cite{liu2019} and our aligned versions.
We train our model to predict using the aligned image then recover the original landmark positions by applying the inverse affine transform to the predicted landmarks.

\begin{figurehere}
    \centering
    \includegraphics[width=19mm, height=19mm]{figs/11125682_1.jpg}\includegraphics[width=19mm, height=19mm]{figs/2521194421_4.jpg} \includegraphics[width=19mm, height=19mm]{figs/11125682_1a.png}\includegraphics[width=19mm, height=19mm]{figs/2521194421_4a.png}
    \\
    \includegraphics[width=19mm, height=19mm]{figs/310508074_1.jpg}\includegraphics[width=19mm, height=19mm]{figs/200767453_0.jpg} \includegraphics[width=19mm, height=19mm]{figs/310508074_1a.png}\includegraphics[width=19mm, height=19mm]{figs/200767453_0a.png}
    \caption{Left: the original images, right: images aligned using the landmarks from the face detector.}
    \label{fig:example_images}
\end{figurehere}


\subsection{Evaluation}

Following \cite{liu2019} we calculate the normalized mean Euclidean error (NME) across all landmarks,
\begin{equation}
    \label{equation:4}
    \text{NME} = \frac{1}{N} \sum_{k=1}^n \frac{||y_k - \hat{y}_k||_2} {d},
\end{equation}
where $n$, $N$, $y$ and $\hat{y}$ are the number of landmarks, number of images, ground truth landmarks and predicted landmarks, respectively.
The normalization factor $d$ is given by $\sqrt{\text{bbox}_w \text{bbox}_h}$ where $\text{bbox}_w$ and $\text{bbox}_h$ are the width and height of the ground truth bounding box, respectively. 
We then compute the cumulative error distribution (CED) curve for images with NME less than $0.08$. 
We define our final evaluation metrics to be the area-under-the-curve (AUC) for the CED curve, the failure rate where the NME is above $0.08$, and the average NME; please refer to \cite{liu2019} for a more detailed description.


For the evaluation dataset, we use the publicly available\footnote{\label{footnote:jd1}\href{https://facial-landmarks-localization-challenge.github.io}{https://facial-landmarks-localization-challenge.github.io}} JD-landmark test set.
These 2000 test images are in the JD-landmark-2 training dataset.
Therefore, we remove these images from JD-landmark-2 to create our training dataset.


\subsection{Baseline settings}

To train our models we use {\sc mxnet}\footnote{\href{https://github.com/apache/incubator-mxnet}{https://github.com/apache/incubator-mxnet}} \citep{chen2015} and {\sc gluoncv} \footnote{\href{https://github.com/dmlc/gluon-cv}{https://github.com/dmlc/gluon-cv}} \citep{guo2020}.
We train our models using ADAM \citep{kingma2014} for 40 epochs with a learning rate of $0.001$ decreasing by an order magnitude at epochs 20 and 30.
All training is done on a single NVIDIA Titan X, using a batch size of 16.
During training, the training dataset is randomly augmented using both the AlexNet style PCA colour augmentation \citep{krizhevsky2012} with $\sigma = 0.05$ and using the {\sc gluoncv} random colour jitter augmentation with a value of $0.4$ for brightness, contrast and saturation.



\section{Results}


\subsection{Baseline results} \label{sec:baseline}
For our baseline, we train MobileNetV2 \citep{sandler2018} with a series of four deconvolutions---all with 256 filters and a stride of two---each followed by a batch normalization layer and a ReLU activation, and finally a 2D convolution with 106 filters to generate the heatmaps.
We set the input size to $192 \times 192 \times 3$, and the generated heatmaps have a size of $96 \times 96$.
Table \ref{tab:1} presents our baseline model results compared to the top three submissions from \cite{liu2019}\textsuperscript{\ref{footnote:jd1}}.
We also experimented with both MobileNet \citep{howard2017} and MobileNetV3 \citep{howard2019} but found they performed significantly worse than MobileNetV2.

\begin{tablehere}
\centering
\begin{tabular}{lccc}
\toprule
Model & AUC (\%) & NME (\%) \\
\midrule
Baidu-VIS$\dagger$ & 84.01 & 1.31 \\
\cite{xiong2020} & 83.34 & 1.35 \\
USTC$\dagger$ & 82.68 & 1.41 \\
VIC$\dagger$ & 82.22 & 1.42 \\
\midrule
MNetV2$_{0.25}$ & 82.20 & 1.42 \\
\bottomrule
\end{tabular}
\newline
\caption{This table presents the performance of our baseline models compared to the top three submissions from \cite{liu2019}$\dagger$ and \cite{xiong2020}.}
\label{tab:1}
\end{tablehere}


\subsection{Downsizing}



\subsubsection*{Pixelshuffle}

To test the performance using pixel-shuffle layers \citep{Shi2016}, we replace our baseline deconvolution layers with four upsampling pixel shuffle blocks.
Each block consists of a 2D convolution, a batch normalization, a ReLU activation, and a final pixel shuffle layer. 
We generate the heatmaps by adding a 2D convolutional layer before the final block.
Our results are presented in Table \ref{tab:shuffle}.

\begin{tablehere}
\centering
\begin{tabular}{lcccc}
\toprule
Strategy & AUC (\%) & GFLOPS & Size (MB) \\
\midrule
\texttt{SSSS} & 79.18 & 0.56 & 6.12 \\
\texttt{DDDD} & 82.20 & 3.50 & 18.26 \\
\bottomrule
\end{tabular}
\newline
\caption{This table presents a comparison between pixel-shuffle and deconvolution upsampling strategies. All models are based on a MobileNetV2$_{0.25}$ backbone and are trained with the baseline settings described in Section \ref{sec:baseline}.}
\label{tab:shuffle}
\end{tablehere}

We find that using pixel-shuffle layers give worse performance than deconvolutions, however, it does significantly reduce the model size and number of FLOPS.
We also experimented using different numbers of upsampling layers but found four to give us the best results for this backbone.




\subsubsection*{Intermittent shuffling}

In the previous section, we compared deconvolution layers and pixel shuffle layers, finding that deconvolutions give better results but are less efficient.
Therefore, we propose a new approach; we combine deconvolution and pixel shuffle layers.
Table \ref{tab:hybrid} shows the results from different layer arrangments, where \texttt{S} and \texttt{D} in the strategy column denote pixel-shuffle and 2D deconvolution layers, respectively.
We notice two things: 
\begin{enumerate}
 \item having the deconvolution layers near the end improves performance,
 \item having pixel shuffle layers after deconvolution layers degrades performance.
\end{enumerate}
For example, \texttt{DDSS} and \texttt{DSDS} are comparable in FLOPS to \texttt{SSSD} and \texttt{SDSD}, however, the latter have far better performance.
Comparing \texttt{DDDD} to \texttt{SDSD} we can see that incorporating pixel-shuffle layers into the upsampling strategy can almost half the number of FLOPS, with only a slight reduction in performance.

\begin{tablehere}
\centering
\begin{tabular}{lcccc}
\toprule
Strategy & AUC (\%) & GFLOPS & Size (MB) \\
\midrule
\texttt{SSSS} & 79.18 & 0.56 & 6.12 \\
\midrule
\texttt{DSSS} & 80.72 & 0.64 & 9.56 \\
\texttt{DDSS} & 80.66 & 0.96 & 12.00 \\
\texttt{DSSD} & 81.55 & 1.10 & 10.08 \\
\texttt{DSDS} & 80.92 & 1.73 & 11.69 \\
\texttt{DDDS} & 80.81 & 2.25 & 16.44 \\
\midrule
\texttt{SSSD} & 81.71 & 1.02 & 6.64 \\
\texttt{SDSD} & 81.90 & 1.29 & 8.76 \\
\texttt{SSDD} & 81.10 & 2.90 & 10.08 \\
\texttt{SDDD} & 81.89 & 3.36 & 13.51 \\
\midrule
\texttt{DDDD} & 82.20 & 3.50 & 18.26 \\
\bottomrule
\end{tabular}
\newline
\caption{This table presents a comparison between our stacked pixel-shuffle and deconvolution upsampling strategies. All models are based on a MobileNetV2$_{0.25}$ backbone and are trained with the baseline settings described in Section \ref{sec:baseline}.}
\label{tab:hybrid}
\end{tablehere}



\subsubsection*{Upsampling filters}

An easy way to reduce the FLOPS is to reduce the number of filters in the deconvolution layers.
We found that reducing the number of filters to 128, using the \texttt{DDDD} upsampling strategy, surprisingly gives us a small boost in AUC of $\sim0.05$ with no increase in failure rate or NME.


\subsection{Upsizing}

By reducing the number of filters and using intermittent shuffling we have drastically reduced the FLOPS of our model. 
Therefore, we can increase the size of our backbone; hopefully resulting in better feature extraction.
Table \ref{tab:finalmnet} presents our results from the final MobileNet models.
We find that using just three upsampling layers consisting of one pixel-shuffle layer and two deconvolutions gives the best performance.

\begin{tablehere}
\centering
\begin{tabular}{lccccc}
\toprule
Backbone       & Strategy      & AUC (\%) & GFLOPS \\
\midrule
MNetV2$_{1.0}$ & \texttt{SSD}  & 84.56   & 0.32 \\
MNetV2$_{1.0}$ & \texttt{SDD}  & 84.67   & 0.43 \\
MNetV2$_{1.0}$ & \texttt{SSSD} & 84.43   & 0.55 \\
MNetV2$_{1.0}$ & \texttt{SDSD} & 84.46   & 0.61 \\
\bottomrule
\end{tabular}
\newline
\caption{This table shows the results for our stacked pixel shuffle and deconvolution models. All these models are trained using the same baseline settings described in Section \ref{sec:baseline}.}
\label{tab:finalmnet}
\end{tablehere}


\section{Final models}


\subsection{Opening our bag of tricks}

To get our final models we use four more tricks.
First, we extend the training dataset by including the horizontal flip of each image and their corresponding landmarks; this resulted in a performance increase of $\sim0.7$ in AUC(\%).
Second, we run inference on a batch containing both the original image and a horizontally flipped copy.
We then average these two predictions, resulting in an AUC(\%) gain of $\sim0.6$.
We also tried to stack the two heatmaps generated for each landmark, then predict using the combined heatmap.
However, we found that this performed almost the same or worse than not using the flipped image.
Third, we employed the random erasing strategy described by \cite{huang2020aid}, resulting in a further AUC(\%) increase of $\sim0.2$.
Last, we incorporate random rotation ($\pm 1^\circ$ -- $\pm 15^\circ$), scaling ($1 \pm 0.05$ -- $\pm 0.2$) and repositioning  ($\pm 5$ -- $\pm 20$ pixels) to each image and it's corresponding landmarks during training.
We found that both random scaling and repositioning result in $\sim1.0$ AUC(\%) performance loss.
Yet, random rotation resulted in a further AUC(\%) increase of $\sim0.2$.
This difference in performance gain/loss indicates that the model is more receptive to consistent face position and scale than rotation.


\subsection{Results on JD-landmark datasets}\label{sec:jd-results}

Table \ref{tab:final} reports our final results on the JD-landmark test set and the JD-landmark-2 validation set.
All of our models perform exceptionally well on the JD-landmark test set, surpassing all the models on the leaderboard\textsuperscript{\ref{footnote:jd1}}.
Our MNetV2$_{1.0}$ is able to rank second on the JD-landmark-2 validation set\textsuperscript{\ref{footnote:jd2}}, while our ResNet18 and ResNet50 are reported to show the upper limit of our approach.

\begin{table*}
\centering
\begin{tabular}{c|lccc}
\toprule
 &     Backbone        & AUC (\%) & Failure Rate (\%) & NME (\%) \\
\midrule
\parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{Challenge 1\textsuperscript{\ref{footnote:jd1}}}}} 
& Our-ResNet50          & 87.06 & 0.00 & 1.03 \\
& Our-ResNet18          & 86.87 & 0.00 & 1.07 \\
& Our-MNetV2$_{1.0}$    & 86.49 & 0.00 & 1.08 \\
& $\dagger$Baidu-VIS    & 84.01 & 0.10 & 1.31 \\
& \cite{xiong2020}      & 83.34 & 0.10 & 1.35 \\
& $\ddagger$USTC & 82.68 & 0.05 & 1.41 \\
\midrule
\parbox[t]{2mm}{\multirow{5}{*}{\rotatebox[origin=c]{90}{Challenge 2\textsuperscript{\ref{footnote:jd2}}}}} 
& Our-ResNet50          & 81.64 & 0.15 & 1.47 \\
& Our-ResNet18          & 81.42 & 0.05 & 1.49 \\
& $\dagger$Sogou AI    & 80.96 & 0.10 & 1.52 \\
& Our-MNetV2$_{1.0}$    & 80.81 & 0.05 & 1.54 \\
& $\ddagger$OPPO Research Institute     & 80.46 & 0.00 & 1.56 \\
\bottomrule
\end{tabular}
\newline
\caption{Top: the evaluation results for different backbones on the JD-landmark test set, bottom: the results for each approach on the JD-landmark-2 validation set; $\dagger$ and $\ddagger$ denote the first and second place entries to each challenge.}
\label{tab:final}
\end{table*}


\subsection{Inference time}\label{sec:inftime}

We report the inference times of our models in \ref{tab:time}. 
To optimize our models we use the Apache TVM\footnote{\href{https://github.com/apache/tvm}{https://github.com/apache/tvm}} compiler framework to optimize our models \citep{tvm2018}.
Our MNetV2$_{1.0}$ can achieve an inference time of 65.70 ms on one core of a desktop CPU (intel i5-9300H) and 4.2 ms on a desktop GPU (NVIDIA GTX 1650).
We also test the inference time of our MNetV2$_{1.0}$ on a mobile device, a Pocophone F1 with a SnapDragon 845 chipset (Kryo 385 CPU and Adreno 630 GPU), achieving an inference time of 67.3 ms.

\begin{tablehere}
\centering
\begin{tabular}{lccc}
\toprule
            & MNetV2$_{1.0}$ & ResNet18 & ResNet50 \\
\midrule
GPU (ms)    & 4.20 & 4.83 & 8.71    \\
CPU (ms)    & 65.70 & 131.5 & 241.5 \\
\midrule
GFLOPS      & 0.43 & 1.13 & 3.23    \\
Size (MB)   & 12.16 & 45.56 & 94.10 \\
\bottomrule
\end{tabular}
\newline
\caption{Inference times for our models measured on an NVIDIA GTX 1650 and an intel i5-9300H, for GPU and CPU, respectively.}
\label{tab:time}
\end{tablehere}




\subsection{Effect on face recognition}

Face recognition systems usually contain four stages: face detection, alignment, embedding and then distance calculation.
In this section, we explore using our landmark localization model to aid the alignment process. 
After we detected the face, we then pass the image to our landmark localization model to obtain the 106 facial landmarks.
Following the conventional five landmarks alignment
\citep{wolf2009,schroff2015,LiuWeiyang2017,wang2018,deng2018,an2020}, we take a subset of five landmarks containing the centres of each eye, the tip of the nose and corners of the mouth.


We evaluate the performance using two different benchmarks for face recognition.
For the first benchmark, we report the face verification accuracy on four public datasets, LFW, CFP-FP, CALFW, CPLFW \citep{lfw,cfp-fp, calfw, cplfw}.
For the second benchmark we follow the IJB-B and IJB-C protocol from ArcFace \citep[see][for more details]{deng2018,an2020}\footnote{\label{footnote:insightface}\href{https://github.com/deepinsight/insightface}{https://github.com/deepinsight/insightface}}, incorporating both detector score and feature normalization.
To detect the faces we use our MNetV2 model from \cite{earp2019}, a publicly avaible ResNet50 model from \citep{deng2019}\textsuperscript{\ref{footnote:insightface}}.
For the face embedding network, we use the pretrained LResNet100E-IR from Insightface\textsuperscript{\ref{footnote:insightface}} and for the landmark model (referred to as 106p) we use our MNetV2$_{1.0}$ presented in Section \ref{sec:jd-results}.
Our results are shown in Table \ref{tab:recognition}.
We give the face detector backbones in the first column, 
 $+106$p indicates that we replace the detector landmarks using the landmark model.


For LFW and CALFW we see little performance change. 
However, for CFP-FP and CPLFW we see accuracy gains of $0.23$ and $0.44$ per cent for MNetV2$_{1.0}$ and $0.07$ and $0.2$ per cent for ResNet50, respectively.
For IJB-B we report the True Acceptance Rate (TAR) at a False Acceptance Rate (FAR) of $1e-4$, finding an improvement of $0.09$ per cent and $0.08$ per cent for MNetV2$_{1.0}$ and ResNet50, respectively. 
Similarly for IJB-C we find a TAR (\@FAR$=1e-4$) improvement of $0.13$ per cent and $0.06$ per cent  for MNetV2$_{1.0}$ and ResNet50, respectively.
We also report the combined inference cost of the face detector and the localization model.
The face detection model is optimized using the same procedure as the landmark model (see Section \ref{sec:inftime}).


\begin{table*}
\centering
\begin{tabular}{lcccc|cc|c}
\toprule
                & LFW   & CFP-FP & CALFW & CPLFW & IJB-B & IJB-C & CPU$_{total}$ (ms) \\
\midrule
MNetV2$_{1.0}$ & 99.87 & 98.36      & \bf{95.82} & 93.58      & 94.81 & 96.14 & 62.66 \\
$\:$ + 106p     & 99.87 & \bf{98.59}      & 95.80 & \bf{94.02}      & \bf{94.90} & \bf{96.27} & 128.4 \\
\midrule
ResNet50        & 99.85      & 98.63       & \bf{95.78} & 93.70      & 94.85      & 96.23      & 261.2 \\
$\:$ + 106p     & \bf{99.87} & \bf{98.70}  & 95.77      & \bf{93.90} & \bf{94.93} & \bf{96.29} & 326.9 \\
\bottomrule
\end{tabular}
\newline
\caption{Face recognition results on benchmark datasets with two different detector backbones: MNetV2$_{1.0}$ and ResNet50, where $+106$p indicates the landmarks have been adjusted by our MNetV2$_{1.0}$ landmark model.}
\label{tab:recognition}
\end{table*}



\section{Conclusions}


We have shown that replacing all the deconvolution layers with pixel-shuffle layers reduces the total number of FLOPS, but this approach significantly impacts the model's performance.
Therefore, we propose the stacked pixel-shuffle and deconvolution upsampling strategy reducing the total number of FLOPS with only a small impact on localization performance.
We applied the random erasing strategy proposed by \cite{huang2020aid} to help improve landmark localization for human joint localization, finding that this approach also affective for facial landmark localization. 
To test the model's dependence on the initial alignment, we randomly rotated, scaled and repositioned each image during training.
We found that both random scaling and repositioning were detrimental to model performance.
On the other hand, random rotation increased the model's performance.
Indicating that the network benefits more from the consistent positioning of our preprocessing method; and is not relying on the alignment.

We present three final models, each with a different backbone, MobileNetV2, ResNet18 and ResNet50.
Our smallest model performs exceptionally well on the JD-landmark-2 validation set, ranking second place only $0.15$ per cent AUC behind first place.
This MobileNetV2 model uses just 43 per cent of the allowed GFLOPS and runs in $65.7$ ms on an intel i5-9300 CPU.
Our ResNet18 surpasses the first place entry by $0.46$ per cent in AUC and $0.03$ per cent in NME, while only falling 13 per cent above the GFLOPS limit and only 13 per cent slower than the MobileNetV2 on GPU.
We also present our ResNet50 version, which only has a slight lead in performance over ResNet18; but is significantly slower.

We have shown adjusting the face landmarks using our landmark localization model leads to improvements in face recognition performance. We see accuracy gains on CFP-FP ($0.07 - 0.23\%$) and CPLFW ($0.2 - 0.44\%$), and also TAR (\@FAR$=1e-4$) for IJB-B ($0.08 - 0.09\%$) and IJB-C ($0.06 - 0.13\%$).
However, we see little change in recognition performance on LFW and CALFW.
One possible cause for this discrepancy is the emphasis both CFP-FP and CPLFW place on using pairs of different poses. 
Therefore, we conclude that either the recognition model is more particular about the alignment of side faces; or that the detector is insufficient to align some of these faces.

Although our models can achieve high accuracy and fast inference speeds, post-processing the landmarks is still expensive.
Future works should direct their attention towards trying to reduce this computational overhead.
For example, \cite{xiong2020} presented a method where the heatmaps are replaced by vectors, drastically reducing post-processing and still achieving high landmark accuracy.
Further research should be continued, in this vein, to break away from the expensive post-processing while maintaining high accuracy.


\section*{Acknowledgements}
We would like to thank Yinglu Liu from JD AI Research for evaluating our models after the deadline had passed. 
We would also like to thank Jeff Hnybida \& Moses Kim for useful conversations during the writing of this paper.


