  We provide a general framework for characterizing the trade-off between
accuracy and robustness in supervised learning. We propose a method and define
quantities to characterize the trade-off between accuracy and robustness for a
given architecture, and provide theoretical insight into the trade-off.
Specifically we introduce a simple trade-off curve, define and study an
influence function that captures the sensitivity, under adversarial attack, of
the optima of a given loss function. We further show how adversarial training
regularizes the parameters in an over-parameterized linear model, recovering
the LASSO and ridge regression as special cases, which also allows us to
theoretically analyze the behavior of the trade-off curve. In experiments, we
demonstrate the corresponding trade-off curves of neural networks and how they
vary with respect to factors such as number of layers, neurons, and across
different network structures. Such information provides a useful guideline to
architecture selection.
