  Text-to-Image (T2I) diffusion models have achieved remarkable success in
synthesizing high-quality images conditioned on text prompts. Recent methods
have tried to replicate the success by either training text-to-video (T2V)
models on a very large number of text-video pairs or adapting T2I models on
text-video pairs independently. Although the latter is computationally less
expensive, it still takes a significant amount of time for per-video adaption.
To address this issue, we propose SAVE, a novel spectral-shift-aware adaptation
framework, in which we fine-tune the spectral shift of the parameter space
instead of the parameters themselves. Specifically, we take the spectral
decomposition of the pre-trained T2I weights and only update the singular
values while freezing the corresponding singular vectors. In addition, we
introduce a spectral shift regularizer aimed at placing tighter constraints on
larger singular values compared to smaller ones. This form of regularization
enables the model to grasp finer details within the video that align with the
provided textual descriptions. We also offer theoretical justification for our
proposed regularization technique. Since we are only dealing with spectral
shifts, the proposed method reduces the adaptation time significantly (approx.
10 times) and has fewer resource constraints for training. Such attributes
posit SAVE to be more suitable for real-world applications, e.g. editing
undesirable content during video streaming. We validate the effectiveness of
SAVE with an extensive experimental evaluation under different settings, e.g.
style transfer, object replacement, privacy preservation, etc.
