  Deep Learning models are highly susceptible to adversarial manipulations that
can lead to catastrophic consequences. One of the most effective methods to
defend against such disturbances is adversarial training but at the cost of
generalization of unseen attacks and transferability across models. In this
paper, we propose a robust defense against adversarial attacks, which is model
agnostic and generalizable to unseen adversaries. Initially, with a baseline
model, we extract the latent representations for each class and adaptively
cluster the latent representations that share a semantic similarity. We obtain
the distributions for the clustered latent representations and from their
originating images, we learn semantic reconstruction dictionaries (SRD). We
adversarially train a new model constraining the latent space representation to
minimize the distance between the adversarial latent representation and the
true cluster distribution. To purify the image, we decompose the input into low
and high-frequency components. The high-frequency component is reconstructed
based on the most adequate SRD from the clean dataset. In order to evaluate the
most adequate SRD, we rely on the distance between robust latent
representations and semantic cluster distributions. The output is a purified
image with no perturbation. Image purification on CIFAR-10 and ImageNet-10
using our proposed method improved the accuracy by more than 10% compared to
state-of-the-art results.
