  The technological advancements in diffusion models (DMs) have demonstrated
unprecedented capabilities in text-to-image generation and are widely used in
diverse applications. However, they have also raised significant societal
concerns, such as the generation of harmful content and copyright disputes.
Machine unlearning (MU) has emerged as a promising solution, capable of
removing undesired generative capabilities from DMs. However, existing MU
evaluation systems present several key challenges that can result in incomplete
and inaccurate assessments. To address these issues, we propose UnlearnCanvas,
a comprehensive high-resolution stylized image dataset that facilitates the
evaluation of the unlearning of artistic styles and associated objects. This
dataset enables the establishment of a standardized, automated evaluation
framework with 7 quantitative metrics assessing various aspects of the
unlearning performance for DMs. Through extensive experiments, we benchmark 9
state-of-the-art MU methods for DMs, revealing novel insights into their
strengths, weaknesses, and underlying mechanisms. Additionally, we explore
challenging unlearning scenarios for DMs to evaluate worst-case performance
against adversarial prompts, the unlearning of finer-scale concepts, and
sequential unlearning. We hope that this study can pave the way for developing
more effective, accurate, and robust DM unlearning methods, ensuring safer and
more ethical applications of DMs in the future. The dataset, benchmark, and
codes are publicly available at https://unlearn-canvas.netlify.app/.
