  Large Language Models (LLMs) have a privacy concern because they memorize
training data (including personally identifiable information (PII) like emails
and phone numbers) and leak it during inference. A company can train an LLM on
its domain-customized data which can potentially also include their users' PII.
In order to comply with privacy laws such as the "right to be forgotten", the
data points of users that are most vulnerable to extraction could be deleted.
We find that once the most vulnerable points are deleted, a new set of points
become vulnerable to extraction. So far, little attention has been given to
understanding memorization for fine-tuned models. In this work, we also show
that not only do fine-tuned models leak their training data but they also leak
the pre-training data (and PII) memorized during the pre-training phase. The
property of new data points becoming vulnerable to extraction after unlearning
and leakage of pre-training data through fine-tuned models can pose significant
privacy and legal concerns for companies that use LLMs to offer services. We
hope this work will start an interdisciplinary discussion within AI and law
communities regarding the need for policies to tackle these issues.
