  The worst-case training principle that minimizes the maximal adversarial
loss, also known as adversarial training (AT), has shown to be a
state-of-the-art approach for enhancing adversarial robustness. Nevertheless,
min-max optimization beyond the purpose of AT has not been rigorously explored
in the adversarial context. In this paper, we show how a general framework of
min-max optimization over multiple domains can be leveraged to advance the
design of different types of adversarial attacks. In particular, given a set of
risk sources, minimizing the worst-case attack loss can be reformulated as a
min-max problem by introducing domain weights that are maximized over the
probability simplex of the domain set. We showcase this unified framework in
three attack generation problems -- attacking model ensembles, devising
universal perturbation under multiple inputs, and crafting attacks resilient to
data transformations. Extensive experiments demonstrate that our approach leads
to substantial attack improvement over the existing heuristic strategies as
well as robustness improvement over state-of-the-art defense methods trained to
be robust against multiple perturbation types. Furthermore, we find that the
self-adjusted domain weights learned from our min-max framework can provide a
holistic tool to explain the difficulty level of attack across domains. Code is
available at https://github.com/wangjksjtu/minmax-adv.
