  Owing to security implications of adversarial vulnerability, adversarial
robustness of deep metric learning models has to be improved. In order to avoid
model collapse due to excessively hard examples, the existing defenses dismiss
the min-max adversarial training, but instead learn from a weak adversary
inefficiently. Conversely, we propose Hardness Manipulation to efficiently
perturb the training triplet till a specified level of hardness for adversarial
training, according to a harder benign triplet or a pseudo-hardness function.
It is flexible since regular training and min-max adversarial training are its
boundary cases. Besides, Gradual Adversary, a family of pseudo-hardness
functions is proposed to gradually increase the specified hardness level during
training for a better balance between performance and robustness. Additionally,
an Intra-Class Structure loss term among benign and adversarial examples
further improves model robustness and efficiency. Comprehensive experimental
results suggest that the proposed method, although simple in its form,
overwhelmingly outperforms the state-of-the-art defenses in terms of
robustness, training efficiency, as well as performance on benign examples.
