  Homomorphic Encryption (HE) is one of the most promising security solutions
to emerging Machine Learning as a Service (MLaaS). Leveled-HE (LHE)-enabled
Convolutional Neural Networks (LHECNNs) are proposed to implement MLaaS to
avoid large bootstrapping overhead. However, prior LHECNNs have to pay
significant computing overhead but achieve only low inference accuracy, due to
their polynomial approximation activations and poolings. Stacking many
polynomial approximation activation layers in a network greatly reduces
inference accuracy, since the polynomial approximation activation errors lead
to a low distortion of the output distribution of the next batch normalization
layer. So the polynomial approximation activations and poolings have become the
obstacle to a fast and accurate LHECNN model.
  In this paper, we propose a Shift-accumulation-based LHE-enabled deep neural
network (SHE) for fast and accurate inferences on encrypted data. We use the
binary-operation-friendly Leveled Fast Homomorphic Encryption over Torus
(LTFHE) encryption scheme to implement ReLU activations and max poolings. We
also adopt the logarithmic quantization to accelerate inferences by replacing
expensive LTFHE multiplications with cheap LTFHE shifts. We propose a mixed
bitwidth accumulator to accelerate accumulations. Since the LTFHE ReLU
activations, max poolings, shifts and accumulations have small multiplicative
depth overhead, SHE can implement much deeper network architectures with more
convolutional and activation layers. Our experimental results show SHE achieves
the state-of-the-art inference accuracy and reduces the inference latency by
76.21% ~ 94.23% over prior LHECNNs on MNIST and CIFAR-10. The source code of
SHE is available at https://github.com/qianlou/SHE.
