  Despite considerable efforts on making them robust, real-world ML-based
systems remain vulnerable to decision based attacks, as definitive proofs of
their operational robustness have so far proven intractable. The canonical
approach in robustness evaluation calls for adaptive attacks, that is with
complete knowledge of the defense and tailored to bypass it. In this study, we
introduce a more expansive notion of being adaptive and show how attacks but
also defenses can benefit by it and by learning from each other through
interaction. We propose and evaluate a framework for adaptively optimizing
black-box attacks and defenses against each other through the competitive game
they form. To reliably measure robustness, it is important to evaluate against
realistic and worst-case attacks. We thus augment both attacks and the evasive
arsenal at their disposal through adaptive control, and observe that the same
can be done for defenses, before we evaluate them first apart and then jointly
under a multi-agent perspective. We demonstrate that active defenses, which
control how the system responds, are a necessary complement to model hardening
when facing decision-based attacks; then how these defenses can be circumvented
by adaptive attacks, only to finally elicit active and adaptive defenses. We
validate our observations through a wide theoretical and empirical
investigation to confirm that AI-enabled adversaries pose a considerable threat
to black-box ML-based systems, rekindling the proverbial arms race where
defenses have to be AI-enabled too. Succinctly, we address the challenges posed
by adaptive adversaries and develop adaptive defenses, thereby laying out
effective strategies in ensuring the robustness of ML-based systems deployed in
the real-world.
