  Machine Learning as a Service (MLaaS) has become a growing trend in recent
years and several such services are currently offered. MLaaS is essentially a
set of services that provides machine learning tools and capabilities as part
of cloud computing services. In these settings, the cloud has pre-trained
models that are deployed and large computing capacity whereas the clients can
use these models to make predictions without having to worry about maintaining
the models and the service. However, the main concern with MLaaS is the privacy
of the client's data.
  Although there have been several proposed approaches in the literature to run
machine learning models on encrypted data, the performance is still far from
being satisfactory for practical use. In this paper, we aim to accelerate the
performance of running machine learning on encrypted data using combination of
Fully Homomorphic Encryption (FHE), Convolutional Neural Networks (CNNs) and
Graphics Processing Units (GPUs). We use a number of optimization techniques,
and efficient GPU-based implementation to achieve high performance. We evaluate
a CNN whose architecture is similar to AlexNet to classify homomorphically
encrypted samples from the Cars Overhead With Context (COWC) dataset. To the
best of our knowledge, it is the first time such a complex network and large
dataset is evaluated on encrypted data. Our approach achieved reasonable
classification accuracy of 95% for the COWC dataset. In terms of performance,
our results show that we could achieve several thousands times speed up when we
implement GPU-accelerated FHE operations on encrypted floating point numbers.
