  Privacy-Preserving Neural Networks (PPNN) are advanced to perform inference
without breaching user privacy, which can serve as an essential tool for
medical diagnosis to simultaneously achieve big data utility and privacy
protection. As one of the key techniques to enable PPNN, Fully Homomorphic
Encryption (FHE) is facing a great challenge that homomorphic operations cannot
be easily adapted for non-linear activation calculations. In this paper,
batch-oriented element-wise data packing and approximate activation are
proposed, which train linear low-degree polynomials to approximate the
non-linear activation function - ReLU. Compared with other approximate
activation methods, the proposed fine-grained, trainable approximation scheme
can effectively reduce the accuracy loss caused by approximation errors.
Meanwhile, due to element-wise data packing, a large batch of images can be
packed and inferred concurrently, leading to a much higher utility ratio of
ciphertext slots. Therefore, although the total inference time increases
sharply, the amortized time for each image actually decreases, especially when
the batch size increases. Furthermore, knowledge distillation is adopted in the
training process to further enhance the inference accuracy. Experiment results
show that when ciphertext inference is performed on 4096 input images, compared
with the current most efficient channel-wise method, the inference accuracy is
improved by 1.65%, and the amortized inference time is reduced by 99.5%.
