  Although motivated by the adaptation of text-to-speech synthesis models, we
argue that more generic parameter-efficient fine-tuning (PEFT) is an
appropriate framework to do such adaptation. However, catastrophic forgetting
remains an issue with PEFT, damaging the pre-trained model's inherent
capabilities. We demonstrate that existing Bayesian learning techniques can be
applied to PEFT to prevent catastrophic forgetting as long as the parameter
shift of the fine-tuned layers can be calculated differentiably. In a
principled series of experiments on language modeling and speech synthesis
tasks, we utilize established Laplace approximations, including diagonal and
Kronecker factored approaches, to regularize PEFT with the low-rank adaptation
(LoRA) and compare their performance in pre-training knowledge preservation.
Our results demonstrate that catastrophic forgetting can be overcome by our
methods without degrading the fine-tuning performance, and using the Kronecker
factored approximations produces a better preservation of the pre-training
knowledge than the diagonal ones.
