  Recent text-to-image generation models have demonstrated impressive
capability of generating text-aligned images with high fidelity. However,
generating images of novel concept provided by the user input image is still a
challenging task. To address this problem, researchers have been exploring
various methods for customizing pre-trained text-to-image generation models.
Currently, most existing methods for customizing pre-trained text-to-image
generation models involve the use of regularization techniques to prevent
over-fitting. While regularization will ease the challenge of customization and
leads to successful content creation with respect to text guidance, it may
restrict the model capability, resulting in the loss of detailed information
and inferior performance. In this work, we propose a novel framework for
customized text-to-image generation without the use of regularization.
Specifically, our proposed framework consists of an encoder network and a novel
sampling method which can tackle the over-fitting problem without the use of
regularization. With the proposed framework, we are able to customize a
large-scale text-to-image generation model within half a minute on single GPU,
with only one image provided by the user. We demonstrate in experiments that
our proposed framework outperforms existing methods, and preserves more
fine-grained details.
