  Adaptive defenses, which optimize at test time, promise to improve
adversarial robustness. We categorize such adaptive test-time defenses, explain
their potential benefits and drawbacks, and evaluate a representative variety
of the latest adaptive defenses for image classification. Unfortunately, none
significantly improve upon static defenses when subjected to our careful case
study evaluation. Some even weaken the underlying static model while
simultaneously increasing inference computation. While these results are
disappointing, we still believe that adaptive test-time defenses are a
promising avenue of research and, as such, we provide recommendations for their
thorough evaluation. We extend the checklist of Carlini et al. (2019) by
providing concrete steps specific to adaptive defenses.
