  Adversarial training has been widely used to enhance the robustness of neural
network models against adversarial attacks. Despite the popularity of neural
network models, a significant gap exists between the natural and robust
accuracy of these models. In this paper, we identify one of the primary reasons
for this gap is the common use of one-hot vectors as labels, which hinders the
learning process for image recognition. Representing ambiguous images with
one-hot vectors is imprecise and may lead the model to suboptimal solutions. To
overcome this issue, we propose a novel method called Low Temperature
Distillation (LTD) that generates soft labels using the modified knowledge
distillation framework. Unlike previous approaches, LTD uses a relatively low
temperature in the teacher model and fixed, but different temperatures for the
teacher and student models. This modification boosts the model's robustness
without encountering the gradient masking problem that has been addressed in
defensive distillation. The experimental results demonstrate the effectiveness
of the proposed LTD method combined with previous techniques, achieving robust
accuracy rates of 58.19%, 31.13%, and 42.08% on CIFAR-10, CIFAR-100, and
ImageNet data sets, respectively, without additional unlabeled data.
