  Recent regulation on right-to-be-forgotten emerges tons of interest in
unlearning pre-trained machine learning models. While approximating a
straightforward yet expensive approach of retrain-from-scratch, recent machine
unlearning methods unlearn a sample by updating weights to remove its influence
on the weight parameters. In this paper, we introduce a simple yet effective
approach to remove a data influence on the deep generative model. Inspired by
works in multi-task learning, we propose to manipulate gradients to regularize
the interplay of influence among samples by projecting gradients onto the
normal plane of the gradients to be retained. Our work is agnostic to
statistics of the removal samples, outperforming existing baselines while
providing theoretical analysis for the first time in unlearning a generative
model.
