  Attack Ensemble (AE), which combines multiple attacks together, provides a
reliable way to evaluate adversarial robustness. In practice, AEs are often
constructed and tuned by human experts, which however tends to be sub-optimal
and time-consuming. In this work, we present AutoAE, a conceptually simple
approach for automatically constructing AEs. In brief, AutoAE repeatedly adds
the attack and its iteration steps to the ensemble that maximizes ensemble
improvement per additional iteration consumed. We show theoretically that
AutoAE yields AEs provably within a constant factor of the optimal for a given
defense. We then use AutoAE to construct two AEs for $l_{\infty}$ and $l_2$
attacks, and apply them without any tuning or adaptation to 45 top adversarial
defenses on the RobustBench leaderboard. In all except one cases we achieve
equal or better (often the latter) robustness evaluation than existing AEs, and
notably, in 29 cases we achieve better robustness evaluation than the best
known one. Such performance of AutoAE shows itself as a reliable evaluation
protocol for adversarial robustness, which further indicates the huge potential
of automatic AE construction. Code is available at
\url{https://github.com/LeegerPENG/AutoAE}.
