  Machine unlearning has raised significant interest with the adoption of laws
ensuring the ``right to be forgotten''. Researchers have provided a
probabilistic notion of approximate unlearning under a similar definition of
Differential Privacy (DP), where privacy is defined as statistical
indistinguishability to retraining from scratch. We propose Langevin
unlearning, an unlearning framework based on noisy gradient descent with
privacy guarantees for approximate unlearning problems. Langevin unlearning
unifies the DP learning process and the privacy-certified unlearning process
with many algorithmic benefits. These include approximate certified unlearning
for non-convex problems, complexity saving compared to retraining, sequential
and batch unlearning for multiple unlearning requests. We verify the
practicality of Langevin unlearning by studying its privacy-utility-complexity
trade-off via experiments on benchmark datasets, and also demonstrate its
superiority against gradient-decent-plus-output-perturbation based approximate
unlearning.
