  Recent advancements in Machine Unlearning (MU) have introduced solutions to
selectively remove certain training samples, such as those with outdated or
sensitive information, from trained models. Despite these advancements,
evaluation of MU methods have been inconsistent, employing different trained
models and architectures, and sample removal strategies, which hampers accurate
comparison. In addition, prior MU approaches have mainly focused on singular
tasks or modalities, which is not comprehensive. To address these limitations,
we develop MU-Bench, the first comprehensive benchmark for MU that (i) unifies
the sets of deleted samples and trained models, and (ii) provides broad
coverage of tasks and data modalities, including previously unexplored domains
such as speech and video classification. Our evaluation show that RandLabel and
SalUn are the most effective general MU approaches on MU-Bench, and BadT and
SCRUB are capable of achieving random performance on the deletion set. We
analyze several under-investigated aspects of unlearning, including
scalability, the impacts of parameter-efficient fine-tuning and curriculum
learning, and susceptibility to dataset biases. MU-Bench provides an
easy-to-use package that includes dataset splits, models, and implementations,
together with a leader board to enable unified and scalable MU research.
