  Machine unlearning strives to uphold the data owners' right to be forgotten
by enabling models to selectively forget specific data. Recent advances suggest
precomputing and storing statistics extracted from second-order information and
implementing unlearning through Newton-style updates. However, the theoretical
analysis of these works often depends on restrictive assumptions of convexity
and smoothness, and those mentioned operations on Hessian matrix are extremely
costly. As a result, applying these works to high-dimensional models becomes
challenging. In this paper, we propose an efficient Hessian-free certified
unlearning. We propose to maintain a statistical vector for each data, computed
through affine stochastic recursion approximation of the difference between
retrained and learned models. Our analysis does not involve inverting Hessian
and thus can be extended to non-convex non-smooth objectives. Under same
assumptions, we demonstrate advancements of proposed method beyond the
state-of-the-art theoretical studies, in terms of generalization, unlearning
guarantee, deletion capacity, and computation/storage complexity, and we show
that the unlearned model of our proposed approach is close to or same as the
retrained model. Based on the strategy of recollecting statistics for
forgetting data, we develop an algorithm that achieves near-instantaneous
unlearning as it only requires a vector addition operation. Experiments
demonstrate that the proposed scheme surpasses existing results by orders of
magnitude in terms of time/storage costs, while also enhancing accuracy.
