  In this paper, a secure Convolutional Neural Network classifier is proposed
using Fully Homomorphic Encryption (FHE). The secure classifier provides a user
with the ability to out-source the computations to a powerful cloud server
and/or setup a server to classify inputs without providing the model or
revealing source data. To this end, a real number framework is developed over
FHE by using a fixed point format with binary digits. This allows for real
number computations for basic operators like addition, subtraction, and
multiplication but also to include secure comparisons and max functions.
Additionally, a rectified linear unit is designed and realized in the
framework. Experimentally, the model was verified using a Convolutional Neural
Network trained for handwritten digits. This encrypted implementation shows
accurate results for all classification when compared against an unencrypted
implementation.
