  Randomized smoothing is a recent and celebrated solution to certify the
robustness of any classifier. While it indeed provides a theoretical robustness
against adversarial attacks, the dimensionality of current classifiers
necessarily imposes Monte Carlo approaches for its application in practice.
This paper questions the effectiveness of randomized smoothing as a defense,
against state of the art black-box attacks. This is a novel perspective, as
previous research works considered the certification as an unquestionable
guarantee. We first formally highlight the mismatch between a theoretical
certification and the practice of attacks on classifiers. We then perform
attacks on randomized smoothing as a defense. Our main observation is that
there is a major mismatch in the settings of the RS for obtaining high
certified robustness or when defeating black box attacks while preserving the
classifier accuracy.
