  Deep neural networks can be exploited using natural adversarial samples,
which do not impact human perception. Current approaches often rely on deep
neural networks' white-box nature to generate these adversarial samples or
synthetically alter the distribution of adversarial samples compared to the
training distribution. In contrast, we propose EvoSeed, a novel evolutionary
strategy-based algorithmic framework for generating photo-realistic natural
adversarial samples. Our EvoSeed framework uses auxiliary Conditional Diffusion
and Classifier models to operate in a black-box setting. We employ CMA-ES to
optimize the search for an initial seed vector, which, when processed by the
Conditional Diffusion Model, results in the natural adversarial sample
misclassified by the Classifier Model. Experiments show that generated
adversarial images are of high image quality, raising concerns about generating
harmful content bypassing safety classifiers. Our research opens new avenues to
understanding the limitations of current safety mechanisms and the risk of
plausible attacks against classifier systems using image generation. Project
Website can be accessed at: https://shashankkotyan.github.io/EvoSeed.
