  Understanding and evaluating the robustness of neural networks under
adversarial settings is a subject of growing interest. Attacks proposed in the
literature usually work with models trained to minimize cross-entropy loss and
output softmax probabilities. In this work, we present interesting experimental
results that suggest the importance of considering other loss functions and
target representations, specifically, (1) training on mean-squared error and
(2) representing targets as codewords generated from a random codebook. We
evaluate the robustness of neural networks that implement these proposed
modifications using existing attacks, showing an increase in accuracy against
untargeted attacks of up to 98.7\% and a decrease of targeted attack success
rates of up to 99.8\%. Our model demonstrates more robustness compared to its
conventional counterpart even against attacks that are tailored to our
modifications. Furthermore, we find that the parameters of our modified model
have significantly smaller Lipschitz bounds, an important measure correlated
with a model's sensitivity to adversarial perturbations.
