  As neural networks revolutionize many applications, significant privacy
conflicts between model users and providers emerge. The cryptography community
developed a variety of techniques for secure computation to address such
privacy issues. As generic techniques for secure computation are typically
prohibitively ineffective, many efforts focus on optimizing their underlying
cryptographic tools. Differently, we propose to optimize the initial design of
crypto-oriented neural architectures and provide a novel Partial Activation
layer. The proposed layer is much faster for secure computation. Evaluating our
method on three state-of-the-art architectures (SqueezeNet, ShuffleNetV2, and
MobileNetV2) demonstrates significant improvement to the efficiency of secure
inference on common evaluation metrics.
