  Hybrid private inference (PI) protocol, which synergistically utilizes both
multi-party computation (MPC) and homomorphic encryption, is one of the most
prominent techniques for PI. However, even the state-of-the-art PI protocols
are bottlenecked by the non-linear layers, especially the activation functions.
Although a standard non-linear activation function can generate higher model
accuracy, it must be processed via a costly garbled-circuit MPC primitive. A
polynomial activation can be processed via Beaver's multiplication triples MPC
primitive but has been incurring severe accuracy drops so far.
  In this paper, we propose an accuracy preserving low-degree polynomial
activation function (AESPA) that exploits the Hermite expansion of the ReLU and
basis-wise normalization. We apply AESPA to popular ML models, such as VGGNet,
ResNet, and pre-activation ResNet, to show an inference accuracy comparable to
those of the standard models with ReLU activation, achieving superior accuracy
over prior low-degree polynomial studies. When applied to the all-RELU baseline
on the state-of-the-art Delphi PI protocol, AESPA shows up to 42.1x and 28.3x
lower online latency and communication cost.
