  Large pretrained language models (LLMs) have shown surprising In-Context
Learning (ICL) ability. An important application in deploying large language
models is to augment LLMs with a private database for some specific task. The
main problem with this promising commercial use is that LLMs have been shown to
memorize their training data and their prompt data are vulnerable to membership
inference attacks (MIA) and prompt leaking attacks. In order to deal with this
problem, we treat LLMs as untrusted in privacy and propose a locally
differentially private framework of in-context learning(LDP-ICL) in the
settings where labels are sensitive. Considering the mechanisms of in-context
learning in Transformers by gradient descent, we provide an analysis of the
trade-off between privacy and utility in such LDP-ICL for classification.
Moreover, we apply LDP-ICL to the discrete distribution estimation problem. In
the end, we perform several experiments to demonstrate our analysis results.
