  Diffusion models have exhibited remarkable prowess in visual generalization.
Building on this success, we introduce an instruction-based object addition
pipeline, named Add-SD, which automatically inserts objects into realistic
scenes with rational sizes and positions. Different from layout-conditioned
methods, Add-SD is solely conditioned on simple text prompts rather than any
other human-costly references like bounding boxes. Our work contributes in
three aspects: proposing a dataset containing numerous instructed image pairs;
fine-tuning a diffusion model for rational generation; and generating synthetic
data to boost downstream tasks. The first aspect involves creating a
RemovalDataset consisting of original-edited image pairs with textual
instructions, where an object has been removed from the original image while
maintaining strong pixel consistency in the background. These data pairs are
then used for fine-tuning the Stable Diffusion (SD) model. Subsequently, the
pretrained Add-SD model allows for the insertion of expected objects into an
image with good rationale. Additionally, we generate synthetic instances for
downstream task datasets at scale, particularly for tail classes, to alleviate
the long-tailed problem. Downstream tasks benefit from the enriched dataset
with enhanced diversity and rationale. Experiments on LVIS val demonstrate that
Add-SD yields an improvement of 4.3 mAP on rare classes over the baseline. Code
and models are available at https://github.com/ylingfeng/Add-SD.
