  Evaluating robustness of machine-learning models to adversarial examples is a
challenging problem. Many defenses have been shown to provide a false sense of
robustness by causing gradient-based attacks to fail, and they have been broken
under more rigorous evaluations. Although guidelines and best practices have
been suggested to improve current adversarial robustness evaluations, the lack
of automatic testing and debugging tools makes it difficult to apply these
recommendations in a systematic manner. In this work, we overcome these
limitations by: (i) categorizing attack failures based on how they affect the
optimization of gradient-based attacks, while also unveiling two novel failures
affecting many popular attack implementations and past evaluations; (ii)
proposing six novel indicators of failure, to automatically detect the presence
of such failures in the attack optimization process; and (iii) suggesting a
systematic protocol to apply the corresponding fixes. Our extensive
experimental analysis, involving more than 15 models in 3 distinct application
domains, shows that our indicators of failure can be used to debug and improve
current adversarial robustness evaluations, thereby providing a first concrete
step towards automatizing and systematizing them. Our open-source code is
available at: https://github.com/pralab/IndicatorsOfAttackFailure.
