  Fine-tuning is a common and effective method for tailoring large language
models (LLMs) to specialized tasks and applications. In this paper, we study
the privacy implications of fine-tuning LLMs on user data. To this end, we
consider a realistic threat model, called user inference, wherein an attacker
infers whether or not a user's data was used for fine-tuning. We design attacks
for performing user inference that require only black-box access to the
fine-tuned LLM and a few samples from a user which need not be from the
fine-tuning dataset. We find that LLMs are susceptible to user inference across
a variety of fine-tuning datasets, at times with near perfect attack success
rates. Further, we theoretically and empirically investigate the properties
that make users vulnerable to user inference, finding that outlier users, users
with identifiable shared features between examples, and users that contribute a
large fraction of the fine-tuning data are most susceptible to attack. Based on
these findings, we identify several methods for mitigating user inference
including training with example-level differential privacy, removing
within-user duplicate examples, and reducing a user's contribution to the
training data. While these techniques provide partial mitigation of user
inference, we highlight the need to develop methods to fully protect fine-tuned
LLMs against this privacy risk.
