  Adversarial attacks can deceive neural networks by adding tiny perturbations
to their input data. Ensemble defenses, which are trained to minimize attack
transferability among sub-models, offer a promising research direction to
improve robustness against such attacks while maintaining a high accuracy on
natural inputs. We discover, however, that recent state-of-the-art (SOTA)
adversarial attack strategies cannot reliably evaluate ensemble defenses,
sizeably overestimating their robustness. This paper identifies the two factors
that contribute to this behavior. First, these defenses form ensembles that are
notably difficult for existing gradient-based method to attack, due to gradient
obfuscation. Second, ensemble defenses diversify sub-model gradients,
presenting a challenge to defeat all sub-models simultaneously, simply summing
their contributions may counteract the overall attack objective; yet, we
observe that ensemble may still be fooled despite most sub-models being
correct. We therefore introduce MORA, a model-reweighing attack to steer
adversarial example synthesis by reweighing the importance of sub-model
gradients. MORA finds that recent ensemble defenses all exhibit varying degrees
of overestimated robustness. Comparing it against recent SOTA white-box
attacks, it can converge orders of magnitude faster while achieving higher
attack success rates across all ensemble models examined with three different
ensemble modes (i.e., ensembling by either softmax, voting or logits). In
particular, most ensemble defenses exhibit near or exactly 0% robustness
against MORA with $\ell^\infty$ perturbation within 0.02 on CIFAR-10, and 0.01
on CIFAR-100. We make MORA open source with reproducible results and
pre-trained models; and provide a leaderboard of ensemble defenses under
various attack strategies.
