  While large pre-trained models have enabled impressive results on a variety
of downstream tasks, the largest existing models still make errors, and even
accurate predictions may become outdated over time. Because detecting all such
failures at training time is impossible, enabling both developers and end users
of such models to correct inaccurate outputs while leaving the model otherwise
intact is desirable. However, the distributed, black-box nature of the
representations learned by large neural networks makes producing such targeted
edits difficult. If presented with only a single problematic input and new
desired output, fine-tuning approaches tend to overfit; other editing
algorithms are either computationally infeasible or simply ineffective when
applied to very large models. To enable easy post-hoc editing at scale, we
propose Model Editor Networks using Gradient Decomposition (MEND), a collection
of small auxiliary editing networks that use a single desired input-output pair
to make fast, local edits to a pre-trained model's behavior. MEND learns to
transform the gradient obtained by standard fine-tuning, using a low-rank
decomposition of the gradient to make the parameterization of this
transformation tractable. MEND can be trained on a single GPU in less than a
day even for 10 billion+ parameter models; once trained MEND enables rapid
application of new edits to the pre-trained model. Our experiments with T5,
GPT, BERT, and BART models show that MEND is the only approach to model editing
that effectively edits the behavior of models with more than 10 billion
parameters. Code and data available at
https://sites.google.com/view/mend-editing.
