  Recently Homomorphic Encryption (HE) is used to implement Privacy-Preserving
Neural Networks (PPNNs) that perform inferences directly on encrypted data
without decryption. Prior PPNNs adopt mobile network architectures such as
SqueezeNet for smaller computing overhead, but we find na\"ively using mobile
network architectures for a PPNN does not necessarily achieve shorter inference
latency. Despite having less parameters, a mobile network architecture
typically introduces more layers and increases the HE multiplicative depth of a
PPNN, thereby prolonging its inference latency. In this paper, we propose a
\textbf{HE}-friendly privacy-preserving \textbf{M}obile neural n\textbf{ET}work
architecture, \textbf{HEMET}. Experimental results show that, compared to
state-of-the-art (SOTA) PPNNs, HEMET reduces the inference latency by
$59.3\%\sim 61.2\%$, and improves the inference accuracy by $0.4 \% \sim
0.5\%$.
