  Deep learning has been able to outperform humans in terms of classification
accuracy in many tasks. However, to achieve robustness to adversarial
perturbations, the best methodologies require to perform adversarial training
on a much larger training set that has been typically augmented using
generative models (e.g., diffusion models). Our main objective in this work, is
to reduce these data requirements while achieving the same or better
accuracy-robustness trade-offs. We focus on data pruning, where some training
samples are removed based on the distance to the model classification boundary
(i.e., margin). We find that the existing approaches that prune samples with
low margin fails to increase robustness when we add a lot of synthetic data,
and explain this situation with a perceptron learning task. Moreover, we find
that pruning high margin samples for better accuracy increases the harmful
impact of mislabeled perturbed data in adversarial training, hurting both
robustness and accuracy. We thus propose PUMA, a new data pruning strategy that
computes the margin using DeepFool, and prunes the training samples of highest
margin without hurting performance by jointly adjusting the training attack
norm on the samples of lowest margin. We show that PUMA can be used on top of
the current state-of-the-art methodology in robustness, and it is able to
significantly improve the model performance unlike the existing data pruning
strategies. Not only PUMA achieves similar robustness with less data, but it
also significantly increases the model accuracy, improving the performance
trade-off.
