  Leveled Homomorphic Encryption (LHE) offers a potential solution that could
allow sectors with sensitive data to utilize the cloud and securely deploy
their models for remote inference with Deep Neural Networks (DNN). However,
this application faces several obstacles due to the limitations of LHE. One of
the main problems is the incompatibility of commonly used nonlinear functions
in DNN with the operations supported by LHE, i.e. addition and multiplication.
As common in LHE approaches, we train a model with a nonlinear function, and
replace it with a low-degree polynomial approximation at inference time on
private data. While this typically leads to approximation errors and loss in
prediction accuracy, we propose a method that reduces this loss to small values
or eliminates it entirely, depending on simple hyper-parameters. This is
achieved by the introduction of a novel and elegantly simple Min-Max
normalization scheme, which scales inputs to nonlinear functions into ranges
with low approximation error. While being intuitive in its concept and trivial
to implement, we empirically show that it offers a stable and effective
approximation solution to nonlinear functions in DNN. In return, this can
enable deeper networks with LHE, and facilitate the development of security-
and privacy-aware analytics applications.
