  Although promising, existing defenses against query-based attacks share a
common limitation: they offer increased robustness against attacks at the price
of a considerable accuracy drop on clean samples. In this work, we show how to
efficiently establish, at test-time, a solid tradeoff between robustness and
accuracy when mitigating query-based attacks. Given that these attacks
necessarily explore low-confidence regions, our insight is that activating
dedicated defenses, such as random noise defense and random image
transformations, only for low-confidence inputs is sufficient to prevent them.
Our approach is independent of training and supported by theory. We verify the
effectiveness of our approach for various existing defenses by conducting
extensive experiments on CIFAR-10, CIFAR-100, and ImageNet. Our results confirm
that our proposal can indeed enhance these defenses by providing better
tradeoffs between robustness and accuracy when compared to state-of-the-art
approaches while being completely training-free.
