  Deep Neural Network based classifiers are known to be vulnerable to
perturbations of inputs constructed by an adversarial attack to force
misclassification. Most studies have focused on how to make vulnerable noise by
gradient based attack methods or to defense model from adversarial attack. The
use of the denoiser model is one of a well-known solution to reduce the
adversarial noise although classification performance had not significantly
improved. In this study, we aim to analyze the propagation of adversarial
attack as an explainable AI(XAI) point of view. Specifically, we examine the
trend of adversarial perturbations through the CNN architectures. To analyze
the propagated perturbation, we measured normalized Euclidean Distance and
cosine distance in each CNN layer between the feature map of the perturbed
image passed through denoiser and the non-perturbed original image. We used
five well-known CNN based classifiers and three gradient-based adversarial
attacks. From the experimental results, we observed that in most cases,
Euclidean Distance explosively increases in the final fully connected layer
while cosine distance fluctuated and disappeared at the last layer. This means
that the use of denoiser can decrease the amount of noise. However, it failed
to defense accuracy degradation.
