  In the literature on adversarial examples, white box and black box attacks
have received the most attention. The adversary is assumed to have either full
(white) or no (black) access to the defender's model. In this work, we focus on
the equally practical gray box setting, assuming an attacker has partial
information. We propose a novel defense that assumes everything but a private
key will be made available to the attacker. Our framework uses an image
denoising procedure coupled with encryption via a discretized Baker map.
Extensive testing against adversarial images (e.g. FGSM, PGD) crafted using
various gradients shows that our defense achieves significantly better results
on CIFAR-10 and CIFAR-100 than the state-of-the-art gray box defenses in both
natural and adversarial accuracy.
