  The emergence of ChatGPT marks the arrival of the large language model (LLM)
era. While LLMs demonstrate their power in a variety of fields, they also raise
serious privacy concerns as the users' queries are sent to the model provider.
On the other side, deploying the LLM on the user's device will also leak all
the model data. Existing methods based on secure multiparty computation (MPC)
managed to protect both the privacy of the model parameters and user queries.
However, they require gigabytes of data transfer and several minutes to
generate just one token, making them impractical for most real-world
applications. To improve the efficiency of private LLM inference, we propose
PermLLM, which accelerates the evaluation of non-linear functions using secure
random permutation. Along with the optimized secret sharing protocols and
homomorphic encryption, PermLLM achieves two-party private inference of the
ChatGLM-6B model at the speed of around 3s/token, under a realistic network
setting (10ms RTT and 1Gbps bandwidth), which is magnitudes faster than
existing MPC solutions.
