  In this paper, we present a practical solution to implement
privacy-preserving CNN training based on mere Homomorphic Encryption (HE)
technique. To our best knowledge, this is the first attempt successfully to
crack this nut and no work ever before has achieved this goal. Several
techniques combine to accomplish the task:: (1) with transfer learning,
privacy-preserving CNN training can be reduced to homomorphic neural network
training, or even multiclass logistic regression (MLR) training; (2) via a
faster gradient variant called $\texttt{Quadratic Gradient}$, an enhanced
gradient method for MLR with a state-of-the-art performance in convergence
speed is applied in this work to achieve high performance; (3) we employ the
thought of transformation in mathematics to transform approximating Softmax
function in the encryption domain to the approximation of the Sigmoid function.
A new type of loss function termed $\texttt{Squared Likelihood Error}$ has been
developed alongside to align with this change.; and (4) we use a simple but
flexible matrix-encoding method named $\texttt{Volley Revolver}$ to manage the
data flow in the ciphertexts, which is the key factor to complete the whole
homomorphic CNN training. The complete, runnable C++ code to implement our work
can be found at:
\href{https://github.com/petitioner/HE.CNNtraining}{$\texttt{https://github.com/petitioner/HE.CNNtraining}$}.
We select $\texttt{REGNET\_X\_400MF}$ as our pre-trained model for transfer
learning. We use the first 128 MNIST training images as training data and the
whole MNIST testing dataset as the testing data. The client only needs to
upload 6 ciphertexts to the cloud and it takes $\sim 21$ mins to perform 2
iterations on a cloud with 64 vCPUs, resulting in a precision of $21.49\%$.
