  Fully Homomorphic Encryption (FHE) has the potential to substantially improve
privacy and security by enabling computation on encrypted data. This is
especially true with deep learning, as today many popular user services are
powered by neural networks. One of the major challenges facing wide-scale
deployment of FHE-secured neural inference is effectively mapping them to the
FHE domain. FHE poses many programming challenges including packing large
vectors, handling expensive rotations, and correctly implementing complex
strided convolutions. This makes programming FHE inferences prone to poor
performance and errors. In this paper we overcome these challenges with Orion,
an automated optimizing FHE compiler for neural inference. Orion automatically
maps PyTorch-specified networks to FHE, handling common layer types and
arbitrary tensor shapes and strides. Moreover, we develop novel optimizations
that balance dense FHE vector packing, efficient rotations, and minimize
operations to improve performance. We have implemented Orion, which will be
open sourced, and evaluated it on common benchmarks used by the FHE deep
learning community. We compare Orion to multiple state-of-the-art solutions and
report iso-accuracy speedups ranging from 2.7$\times$ to 20.5$\times$.
