  Machine unlearning methods seek to revise pretrained models such that effects
of certain training samples can be removed. In addition to effective erasure,
low computational cost and general utility retention are also highly desirable.
Existing unlearning methods usually involve iterative updates over the model
parameters, which incurs a high computational cost. In this work, we propose an
efficient method that only requires a one-time gradient computation, with which
we modify only a single layer of model parameters. Specifically, we first
identify a small number of model layers that lie on the Pareto front of high
forget importance and low retain influence as critical layers. Then we search
for a suitable step size and take a step along the gradient direction of a
single critical layer while keeping other layers frozen. This method is highly
modular and can be used to unlearn multiple concepts simultaneously in a
controllable manner. We demonstrate the effectiveness and efficiency of this
method on various models including CLIP, stable diffusion, and VLMs, surpassing
other state-of-the-art methods.
