  Machine unlearning is motivated by desire for data autonomy: a person can
request to have their data's influence removed from deployed models, and those
models should be updated as if they were retrained without the person's data.
We show that, counter-intuitively, these updates expose individuals to
high-accuracy reconstruction attacks which allow the attacker to recover their
data in its entirety, even when the original models are so simple that privacy
risk might not otherwise have been a concern. We show how to mount a
near-perfect attack on the deleted data point from linear regression models. We
then generalize our attack to other loss functions and architectures, and
empirically demonstrate the effectiveness of our attacks across a wide range of
datasets (capturing both tabular and image data). Our work highlights that
privacy risk is significant even for extremely simple model classes when
individuals can request deletion of their data from the model.
