  Enabling private inference is crucial for many cloud inference services that
are based on Transformer models. However, existing private inference solutions
can increase the inference latency by more than 60x or significantly compromise
the inference quality. In this paper, we design the framework MPCFORMER as a
practical solution, using Secure Multi-Party Computation (MPC) and Knowledge
Distillation (KD). Through extensive evaluations, we show that MPCFORMER
significantly speeds up Transformer inference in MPC settings while achieving
similar ML performance to the input model. On the IMDb dataset, it achieves
similar performance to BERTBASE, while being 5.3x faster. On the GLUE
benchmark, it achieves 97% performance of BERTBASE with a 2.2x speedup.
MPCFORMER remains effective with different trained Transformer weights such as
ROBERTABASE and larger models including BERTLarge. Code is available at
https://github.com/MccRee177/MPCFormer.
