  Machine unlearning is the process of updating machine learning models to
remove the information of specific training data samples, in order to comply
with data protection regulations that allow individuals to request the removal
of their personal data. Despite the recent development of numerous unlearning
algorithms, reliable evaluation of these algorithms remains an open research
question. In this work, we focus on membership inference attack (MIA) based
evaluation, one of the most common approaches for evaluating unlearning
algorithms, and address various pitfalls of existing evaluation metrics that
lack reliability. Specifically, we propose a game-theoretic framework that
formalizes the evaluation process as a game between unlearning algorithms and
MIA adversaries, measuring the data removal efficacy of unlearning algorithms
by the capability of the MIA adversaries. Through careful design of the game,
we demonstrate that the natural evaluation metric induced from the game enjoys
provable guarantees that the existing evaluation metrics fail to satisfy.
Furthermore, we propose a practical and efficient algorithm to estimate the
evaluation metric induced from the game, and demonstrate its effectiveness
through both theoretical analysis and empirical experiments. This work presents
a novel and reliable approach to empirically evaluating unlearning algorithms,
paving the way for the development of more effective unlearning techniques.
