  Generative diffusion models, including Stable Diffusion and Midjourney, can
generate visually appealing, diverse, and high-resolution images for various
applications. These models are trained on billions of internet-sourced images,
raising significant concerns about the potential unauthorized use of
copyright-protected images. In this paper, we examine whether it is possible to
determine if a specific image was used in the training set, a problem known in
the cybersecurity community and referred to as a membership inference attack.
Our focus is on Stable Diffusion, and we address the challenge of designing a
fair evaluation framework to answer this membership question. We propose a
methodology to establish a fair evaluation setup and apply it to Stable
Diffusion, enabling potential extensions to other generative models. Utilizing
this evaluation setup, we execute membership attacks (both known and newly
introduced). Our research reveals that previously proposed evaluation setups do
not provide a full understanding of the effectiveness of membership inference
attacks. We conclude that the membership inference attack remains a significant
challenge for large diffusion models (often deployed as black-box systems),
indicating that related privacy and copyright issues will persist in the
foreseeable future.
