  Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have
societal implications that extend beyond the field of computer science. These
models require large image databases like LAION-2B, which contain two billion
images. At this scale, manual inspection is difficult and automated analysis is
challenging. In addition, recent studies show that duplicated images pose
copyright problems for models trained on LAION2B, which hinders its usability.
This paper proposes an algorithmic chain that runs with modest compute, that
compresses CLIP features to enable efficient duplicate detection, even for vast
image volumes. Our approach demonstrates that roughly 700 million images, or
about 30\%, of LAION-2B's images are likely duplicated. Our method also
provides the histograms of duplication on this dataset, which we use to reveal
more examples of verbatim copies by Stable Diffusion and further justify the
approach. The current version of the de-duplicated set will be distributed
online.
