  Machine learning on encrypted data has received a lot of attention thanks to
recent breakthroughs in homomorphic encryption and secure multi-party
computation. It allows outsourcing computation to untrusted servers without
sacrificing privacy of sensitive data. We propose a practical framework to
perform partially encrypted and privacy-preserving predictions which combines
adversarial training and functional encryption. We first present a new
functional encryption scheme to efficiently compute quadratic functions so that
the data owner controls what can be computed but is not involved in the
calculation: it provides a decryption key which allows one to learn a specific
function evaluation of some encrypted data. We then show how to use it in
machine learning to partially encrypt neural networks with quadratic activation
functions at evaluation time, and we provide a thorough analysis of the
information leaks based on indistinguishability of data items of the same
label. Last, since most encryption schemes cannot deal with the last
thresholding operation used for classification, we propose a training method to
prevent selected sensitive features from leaking, which adversarially optimizes
the network against an adversary trying to identify these features. This is
interesting for several existing works using partially encrypted machine
learning as it comes with little reduction on the model's accuracy and
significantly improves data privacy.
