  Empirical robustness evaluation (RE) of deep learning models against
adversarial perturbations entails solving nontrivial constrained optimization
problems. Existing numerical algorithms that are commonly used to solve them in
practice predominantly rely on projected gradient, and mostly handle
perturbations modeled by the $\ell_1$, $\ell_2$ and $\ell_\infty$ distances. In
this paper, we introduce a novel algorithmic framework that blends a
general-purpose constrained-optimization solver PyGRANSO with Constraint
Folding (PWCF), which can add more reliability and generality to the
state-of-the-art RE packages, e.g., AutoAttack. Regarding reliability, PWCF
provides solutions with stationarity measures and feasibility tests to assess
the solution quality. For generality, PWCF can handle perturbation models that
are typically inaccessible to the existing projected gradient methods; the main
requirement is the distance metric to be almost everywhere differentiable.
Taking advantage of PWCF and other existing numerical algorithms, we further
explore the distinct patterns in the solutions found for solving these
optimization problems using various combinations of losses, perturbation
models, and optimization algorithms. We then discuss the implications of these
patterns on the current robustness evaluation and adversarial training.
