  Convolutional neural networks (CNNs) have enabled significant performance
leaps in medical image classification tasks. However, translating neural
network models for clinical applications remains challenging due to data
privacy issues. Fully Homomorphic Encryption (FHE) has the potential to address
this challenge as it enables the use of CNNs on encrypted images. However,
current HE technology poses immense computational and memory overheads,
particularly for high-resolution images such as those seen in the clinical
context. We present CaRENets: Compact and Resource-Efficient CNNs for high
performance and resource-efficient inference on high-resolution encrypted
images in practical applications. At the core, CaRENets comprises a new FHE
compact packing scheme that is tightly integrated with CNN functions. CaRENets
offers dual advantages of memory efficiency (due to compact packing of images
and CNN activations) and inference speed (due to the reduction in the number of
ciphertexts created and the associated mathematical operations) over standard
interleaved packing schemes. We apply CaRENets to perform homomorphic
abnormality detection with 80-bit security level in two clinical conditions -
Retinopathy of Prematurity (ROP) and Diabetic Retinopathy (DR). The ROP dataset
comprises 96 x 96 grayscale images, while the DR dataset comprises 256 x 256
RGB images. We demonstrate over 45x improvement in memory efficiency and 4-5x
speedup in inference over the interleaved packing schemes. As our approach
enables memory-efficient low-latency HE inference without imposing additional
communication burden, it has implications for practical and secure deep
learning inference in clinical imaging.
