  Machine learning models are vulnerable to adversarial attacks, including
attacks that leak information about the model's training data. There has
recently been an increase in interest about how to best address privacy
concerns, especially in the presence of data-removal requests. Machine
unlearning algorithms aim to efficiently update trained models to comply with
data deletion requests while maintaining performance and without having to
resort to retraining the model from scratch, a costly endeavor. Several
algorithms in the machine unlearning literature demonstrate some level of
privacy gains, but they are often evaluated only on rudimentary membership
inference attacks, which do not represent realistic threats. In this paper we
describe and propose alternative evaluation methods for three key shortcomings
in the current evaluation of unlearning algorithms. We show the utility of our
alternative evaluations via a series of experiments of state-of-the-art
unlearning algorithms on different computer vision datasets, presenting a more
detailed picture of the state of the field.
