  Recent work has proposed stateful defense models (SDMs) as a compelling
strategy to defend against a black-box attacker who only has query access to
the model, as is common for online machine learning platforms. Such stateful
defenses aim to defend against black-box attacks by tracking the query history
and detecting and rejecting queries that are "similar" and thus preventing
black-box attacks from finding useful gradients and making progress towards
finding adversarial attacks within a reasonable query budget. Recent SDMs
(e.g., Blacklight and PIHA) have shown remarkable success in defending against
state-of-the-art black-box attacks. In this paper, we show that SDMs are highly
vulnerable to a new class of adaptive black-box attacks. We propose a novel
adaptive black-box attack strategy called Oracle-guided Adaptive Rejection
Sampling (OARS) that involves two stages: (1) use initial query patterns to
infer key properties about an SDM's defense; and, (2) leverage those extracted
properties to design subsequent query patterns to evade the SDM's defense while
making progress towards finding adversarial inputs. OARS is broadly applicable
as an enhancement to existing black-box attacks - we show how to apply the
strategy to enhance six common black-box attacks to be more effective against
current class of SDMs. For example, OARS-enhanced versions of black-box attacks
improved attack success rate against recent stateful defenses from almost 0% to
to almost 100% for multiple datasets within reasonable query budgets.
