  Designing powerful adversarial attacks is of paramount importance for the
evaluation of $\ell_p$-bounded adversarial defenses. Projected Gradient Descent
(PGD) is one of the most effective and conceptually simple algorithms to
generate such adversaries. The search space of PGD is dictated by the steepest
ascent directions of an objective. Despite the plethora of objective function
choices, there is no universally superior option and robustness overestimation
may arise from ill-suited objective selection. Driven by this observation, we
postulate that the combination of different objectives through a simple loss
alternating scheme renders PGD more robust towards design choices. We
experimentally verify this assertion on a synthetic-data example and by
evaluating our proposed method across 25 different $\ell_{\infty}$-robust
models and 3 datasets. The performance improvement is consistent, when compared
to the single loss counterparts. In the CIFAR-10 dataset, our strongest
adversarial attack outperforms all of the white-box components of AutoAttack
(AA) ensemble, as well as the most powerful attacks existing on the literature,
achieving state-of-the-art results in the computational budget of our study
($T=100$, no restarts).
