  The practical needs of the ``right to be forgotten'' and poisoned data
removal call for efficient \textit{machine unlearning} techniques, which enable
machine learning models to unlearn, or to forget a fraction of training data
and its lineage. Recent studies on machine unlearning for deep neural networks
(DNNs) attempt to destroy the influence of the forgetting data by scrubbing the
model parameters. However, it is prohibitively expensive due to the large
dimension of the parameter space. In this paper, we refocus our attention from
the parameter space to the decision space of the DNN model, and propose
Boundary Unlearning, a rapid yet effective way to unlearn an entire class from
a trained DNN model. The key idea is to shift the decision boundary of the
original DNN model to imitate the decision behavior of the model retrained from
scratch. We develop two novel boundary shift methods, namely Boundary Shrink
and Boundary Expanding, both of which can rapidly achieve the utility and
privacy guarantees. We extensively evaluate Boundary Unlearning on CIFAR-10 and
Vggface2 datasets, and the results show that Boundary Unlearning can
effectively forget the forgetting class on image classification and face
recognition tasks, with an expected speed-up of $17\times$ and $19\times$,
respectively, compared with retraining from the scratch.
