  We present SEALion: an extensible framework for privacy-preserving machine
learning with homomorphic encryption. It allows one to learn deep neural
networks that can be seamlessly utilized for prediction on encrypted data. The
framework consists of two layers: the first is built upon TensorFlow and SEAL
and exposes standard algebra and deep learning primitives; the second
implements a Keras-like syntax for training and inference with neural networks.
Given a required level of security, a user is abstracted from the details of
the encoding and the encryption scheme, allowing quick prototyping. We present
two applications that exemplifying the extensibility of our proposal, which are
also of independent interest: i) improving efficiency of neural network
inference by an activity sparsifier and ii) transfer learning by querying a
server-side Variational AutoEncoder that can handle encrypted data.
