  Homomorphic encryption (HE) enables privacy-preserving deep learning by
allowing computations on encrypted data without decryption. However, deploying
convolutional neural networks (CNNs) with HE is challenging due to the need to
convert input data into a two-dimensional matrix for convolution using the
im2col technique, which rearranges the input for efficient computation. This
restricts the types of CNN models that can be used since the encrypted data
structure must be compatible with the specific model. UniHENN is a novel
HE-based CNN architecture that eliminates the need for im2col, enhancing its
versatility and compatibility with a broader range of CNN models. UniHENN
flattens input data to one dimension without using im2col. The kernel performs
convolutions by traversing the image, using incremental rotations and
structured multiplication on the flattened input, with results spaced by the
stride interval. Experimental results show that UniHENN significantly
outperforms the state-of-the-art 2D CNN inference architecture named PyCrCNN in
terms of inference time. For example, on the LeNet-1 model, UniHENN achieves an
average inference time of 30.089 seconds, about 26.6 times faster than
PyCrCNN's 800.591 seconds. Furthermore, UniHENN outperforms TenSEAL, an
im2col-optimized CNN model, in concurrent image processing. For ten samples,
UniHENN (16.247 seconds) was about 3.9 times faster than TenSEAL (63.706
seconds), owing to its support for batch processing of up to 10 samples. We
demonstrate UniHENN's adaptability to various CNN architectures, including a 1D
CNN and six 2D CNNs, highlighting its flexibility and efficiency for
privacy-preserving cloud-based CNN services.
