  Pre-trained large text-to-image (T2I) models with an appropriate text prompt
has attracted growing interests in customized images generation field. However,
catastrophic forgetting issue make it hard to continually synthesize new
user-provided styles while retaining the satisfying results amongst learned
styles. In this paper, we propose MuseumMaker, a method that enables the
synthesis of images by following a set of customized styles in a never-end
manner, and gradually accumulate these creative artistic works as a Museum.
When facing with a new customization style, we develop a style distillation
loss module to extract and learn the styles of the training data for new image
generation. It can minimize the learning biases caused by content of new
training images, and address the catastrophic overfitting issue induced by
few-shot images. To deal with catastrophic forgetting amongst past learned
styles, we devise a dual regularization for shared-LoRA module to optimize the
direction of model update, which could regularize the diffusion model from both
weight and feature aspects, respectively. Meanwhile, to further preserve
historical knowledge from past styles and address the limited representability
of LoRA, we consider a task-wise token learning module where a unique token
embedding is learned to denote a new style. As any new user-provided style
come, our MuseumMaker can capture the nuances of the new styles while
maintaining the details of learned styles. Experimental results on diverse
style datasets validate the effectiveness of our proposed MuseumMaker method,
showcasing its robustness and versatility across various scenarios.
