  ML-as-a-service continues to grow, and so does the need for very strong
privacy guarantees. Secure inference has emerged as a potential solution,
wherein cryptographic primitives allow inference without revealing users'
inputs to a model provider or model's weights to a user. For instance, the
model provider could be a diagnostics company that has trained a
state-of-the-art DenseNet-121 model for interpreting a chest X-ray and the user
could be a patient at a hospital. While secure inference is in principle
feasible for this setting, there are no existing techniques that make it
practical at scale. The CrypTFlow2 framework provides a potential solution with
its ability to automatically and correctly translate clear-text inference to
secure inference for arbitrary models. However, the resultant secure inference
from CrypTFlow2 is impractically expensive: Almost 3TB of communication is
required to interpret a single X-ray on DenseNet-121. In this paper, we address
this outstanding challenge of inefficiency of secure inference with three
contributions. First, we show that the primary bottlenecks in secure inference
are large linear layers which can be optimized with the choice of network
backbone and the use of operators developed for efficient clear-text inference.
This finding and emphasis deviates from many recent works which focus on
optimizing non-linear activation layers when performing secure inference of
smaller networks. Second, based on analysis of a bottle-necked convolution
layer, we design a X-operator which is a more efficient drop-in replacement.
Third, we show that the fast Winograd convolution algorithm further improves
efficiency of secure inference. In combination, these three optimizations prove
to be highly effective for the problem of X-ray interpretation trained on the
CheXpert dataset.
