  There is a rapidly growing interest in controlling consistency across
multiple generated images using diffusion models. Among various methods, recent
works have found that simply manipulating attention modules by concatenating
features from multiple reference images provides an efficient approach to
enhancing consistency without fine-tuning. Despite its popularity and success,
few studies have elucidated the underlying mechanisms that contribute to its
effectiveness. In this work, we reveal that the popular approach is a linear
interpolation of image self-attention and cross-attention between synthesized
content and reference features, with a constant rank-1 coefficient. Motivated
by this observation, we find that a rank-1 coefficient is not necessary and
simplifies the controllable generation mechanism. The resulting algorithm,
which we coin as RefDrop, allows users to control the influence of reference
context in a direct and precise manner. Besides further enhancing consistency
in single-subject image generation, our method also enables more interesting
applications, such as the consistent generation of multiple subjects,
suppressing specific features to encourage more diverse content, and
high-quality personalized video generation by boosting temporal consistency.
Even compared with state-of-the-art image-prompt-based generators, such as
IP-Adapter, RefDrop is competitive in terms of controllability and quality
while avoiding the need to train a separate image encoder for feature injection
from reference images, making it a versatile plug-and-play solution for any
image or video diffusion model.
