  The growth of the Machine-Learning-As-A-Service (MLaaS) market has
highlighted clients' data privacy and security issues. Private inference (PI)
techniques using cryptographic primitives offer a solution but often have high
computation and communication costs, particularly with non-linear operators
like ReLU. Many attempts to reduce ReLU operations exist, but they may need
heuristic threshold selection or cause substantial accuracy loss. This work
introduces AutoReP, a gradient-based approach to lessen non-linear operators
and alleviate these issues. It automates the selection of ReLU and polynomial
functions to speed up PI applications and introduces distribution-aware
polynomial approximation (DaPa) to maintain model expressivity while accurately
approximating ReLUs. Our experimental results demonstrate significant accuracy
improvements of 6.12% (94.31%, 12.9K ReLU budget, CIFAR-10), 8.39% (74.92%,
12.9K ReLU budget, CIFAR-100), and 9.45% (63.69%, 55K ReLU budget,
Tiny-ImageNet) over current state-of-the-art methods, e.g., SNL. Morever,
AutoReP is applied to EfficientNet-B2 on ImageNet dataset, and achieved 75.55%
accuracy with 176.1 times ReLU budget reduction.
