  In practice, metric analysis on a specific train and test dataset does not
guarantee reliable or fair ML models. This is partially due to the fact that
obtaining a balanced, diverse, and perfectly labeled dataset is typically
expensive, time-consuming, and error-prone. Rather than relying on a carefully
designed test set to assess ML models' failures, fairness, or robustness, this
paper proposes Semantic Image Attack (SIA), a method based on the adversarial
attack that provides semantic adversarial images to allow model diagnosis,
interpretability, and robustness. Traditional adversarial training is a popular
methodology for robustifying ML models against attacks. However, existing
adversarial methods do not combine the two aspects that enable the
interpretation and analysis of the model's flaws: semantic traceability and
perceptual quality. SIA combines the two features via iterative gradient ascent
on a predefined semantic attribute space and the image space. We illustrate the
validity of our approach in three scenarios for keypoint detection and
classification. (1) Model diagnosis: SIA generates a histogram of attributes
that highlights the semantic vulnerability of the ML model (i.e., attributes
that make the model fail). (2) Stronger attacks: SIA generates adversarial
examples with visually interpretable attributes that lead to higher attack
success rates than baseline methods. The adversarial training on SIA improves
the transferable robustness across different gradient-based attacks. (3)
Robustness to imbalanced datasets: we use SIA to augment the underrepresented
classes, which outperforms strong augmentation and re-balancing baselines.
