  Machine unlearning, an emerging research topic focusing on compliance with
data privacy regulations, enables trained models to remove the information
learned from specific data. While many existing methods indirectly address this
issue by intentionally injecting incorrect supervisions, they can drastically
and unpredictably alter the decision boundaries and feature spaces, leading to
training instability and undesired side effects. To fundamentally approach this
task, we first analyze the changes in latent feature spaces between original
and retrained models, and observe that the feature representations of samples
not involved in training are closely aligned with the feature manifolds of
previously seen samples in training. Based on these findings, we introduce a
novel evaluation metric for machine unlearning, coined dimensional alignment,
which measures the alignment between the eigenspaces of the forget and retain
set samples. We employ this metric as a regularizer loss to build a robust and
stable unlearning framework, which is further enhanced by integrating a
self-distillation loss and an alternating training scheme. Our framework
effectively eliminates information from the forget set and preserves knowledge
from the retain set. Lastly, we identify critical flaws in established
evaluation metrics for machine unlearning, and introduce new evaluation tools
that more accurately reflect the fundamental goals of machine unlearning.
