  Deep neural networks are vulnerable to adversarial attacks. Recent studies
about adversarial robustness focus on the loss landscape in the parameter space
since it is related to optimization and generalization performance. These
studies conclude that the difficulty of adversarial training is caused by the
non-smoothness of the loss function: i.e., its gradient is not Lipschitz
continuous. However, this analysis ignores the dependence of adversarial
attacks on model parameters. Since adversarial attacks are optimized for
models, they should depend on the parameters. Considering this dependence, we
analyze the smoothness of the loss function of adversarial training using the
optimal attacks for the model parameter in more detail. We reveal that the
constraint of adversarial attacks is one cause of the non-smoothness and that
the smoothness depends on the types of the constraints. Specifically, the
$L_\infty$ constraint can cause non-smoothness more than the $L_2$ constraint.
Moreover, our analysis implies that if we flatten the loss function with
respect to input data, the Lipschitz constant of the gradient of adversarial
loss tends to increase. To address the non-smoothness, we show that EntropySGD
smoothens the non-smooth loss and improves the performance of adversarial
training.
