  Machine learning (ML) algorithms are increasingly important for the success
of products and services, especially considering the growing amount and
availability of data. This also holds for areas handling sensitive data, e.g.
applications processing medical data or facial images. However, people are
reluctant to pass their personal sensitive data to a ML service provider. At
the same time, service providers have a strong interest in protecting their
intellectual property and therefore refrain from publicly sharing their ML
model. Fully homomorphic encryption (FHE) is a promising technique to enable
individuals using ML services without giving up privacy and protecting the ML
model of service providers at the same time. Despite steady improvements, FHE
is still hardly integrated in today's ML applications.
  We introduce HE-MAN, an open-source two-party machine learning toolset for
privacy preserving inference with ONNX models and homomorphically encrypted
data. Both the model and the input data do not have to be disclosed. HE-MAN
abstracts cryptographic details away from the users, thus expertise in FHE is
not required for either party. HE-MAN 's security relies on its underlying FHE
schemes. For now, we integrate two different homomorphic encryption schemes,
namely Concrete and TenSEAL. Compared to prior work, HE-MAN supports a broad
range of ML models in ONNX format out of the box without sacrificing accuracy.
We evaluate the performance of our implementation on different network
architectures classifying handwritten digits and performing face recognition
and report accuracy and latency of the homomorphically encrypted inference.
Cryptographic parameters are automatically derived by the tools. We show that
the accuracy of HE-MAN is on par with models using plaintext input while
inference latency is several orders of magnitude higher compared to the
plaintext case.
