  In the big data era, cloud-based machine learning as a service (MLaaS) has
attracted considerable attention. However, when handling sensitive data, such
as financial and medical data, a privacy issue emerges, because the cloud
server can access clients' raw data. A common method of handling sensitive data
in the cloud uses homomorphic encryption, which allows computation over
encrypted data without decryption. Previous research usually adopted a
low-degree polynomial mapping function, such as the square function, for data
classification. However, this technique results in low classification accuracy.
In this study, we seek to improve the classification accuracy for inference
processing in a convolutional neural network (CNN) while using homomorphic
encryption. We adopt an activation function that approximates Google's Swish
activation function while using a fourth-order polynomial. We also adopt batch
normalization to normalize the inputs for the Swish function to fit the input
range to minimize the error. We implemented CNN inference labeling over
homomorphic encryption using the Microsoft's Simple Encrypted Arithmetic
Library for the Cheon-Kim-Kim-Song (CKKS) scheme. The experimental evaluations
confirmed classification accuracies of 99.22% and 80.48% for MNIST and
CIFAR-10, respectively, which entails 0.04% and 4.11% improvements,
respectively, over previous methods.
