  Privacy-preserving neural networks have attracted increasing attention in
recent years, and various algorithms have been developed to keep the balance
between accuracy, computational complexity and information security from the
cryptographic view. This work takes a different view from the input data and
structure of neural networks. We decompose the input data (e.g., some images)
into sensitive and insensitive segments according to importance and privacy.
The sensitive segment includes some important and private information such as
human faces and we take strong homomorphic encryption to keep security, whereas
the insensitive one contains some background and we add perturbations. We
propose the bi-CryptoNets, i.e., plaintext and ciphertext branches, to deal
with two segments, respectively, and ciphertext branch could utilize the
information from plaintext branch by unidirectional connections. We adopt
knowledge distillation for our bi-CryptoNets by transferring representations
from a well-trained teacher neural network. Empirical studies show the
effectiveness and decrease of inference latency for our bi-CryptoNets.
