  Diffusion models have recently demonstrated their effectiveness in generating
extremely high-quality images and are now utilized in a wide range of
applications, including automatic sketch colorization. Although many methods
have been developed for guided sketch colorization, there has been limited
exploration of the potential conflicts between image prompts and sketch inputs,
which can lead to severe deterioration in the results. Therefore, this paper
exhaustively investigates reference-based sketch colorization models that aim
to colorize sketch images using reference color images. We specifically
investigate two critical aspects of reference-based diffusion models: the
"distribution problem", which is a major shortcoming compared to text-based
counterparts, and the capability in zero-shot sequential text-based
manipulation. We introduce two variations of an image-guided latent diffusion
model utilizing different image tokens from the pre-trained CLIP image encoder
and propose corresponding manipulation methods to adjust their results
sequentially using weighted text inputs. We conduct comprehensive evaluations
of our models through qualitative and quantitative experiments as well as a
user study.
