  Large Language Models (LLMs) have shown greatly enhanced performance in
recent years, attributed to increased size and extensive training data. This
advancement has led to widespread interest and adoption across industries and
the public. However, training data memorization in Machine Learning models
scales with model size, particularly concerning for LLMs. Memorized text
sequences have the potential to be directly leaked from LLMs, posing a serious
threat to data privacy. Various techniques have been developed to attack LLMs
and extract their training data. As these models continue to grow, this issue
becomes increasingly critical. To help researchers and policymakers understand
the state of knowledge around privacy attacks and mitigations, including where
more work is needed, we present the first SoK on data privacy for LLMs. We (i)
identify a taxonomy of salient dimensions where attacks differ on LLMs, (ii)
systematize existing attacks, using our taxonomy of dimensions to highlight key
trends, (iii) survey existing mitigation strategies, highlighting their
strengths and limitations, and (iv) identify key gaps, demonstrating open
problems and areas for concern.
