  Correctly evaluating defenses against adversarial examples has proven to be
extremely difficult. Despite the significant amount of recent work attempting
to design defenses that withstand adaptive attacks, few have succeeded; most
papers that propose defenses are quickly shown to be incorrect.
  We believe a large contributing factor is the difficulty of performing
security evaluations. In this paper, we discuss the methodological foundations,
review commonly accepted best practices, and suggest new methods for evaluating
defenses to adversarial examples. We hope that both researchers developing
defenses as well as readers and reviewers who wish to understand the
completeness of an evaluation consider our advice in order to avoid common
pitfalls.
