  Certified defenses against small-norm adversarial examples have received
growing attention in recent years; though certified accuracies of
state-of-the-art methods remain far below their non-robust counterparts,
despite the fact that benchmark datasets have been shown to be well-separated
at far larger radii than the literature generally attempts to certify. In this
work, we offer insights that identify potential factors in this performance
gap. Specifically, our analysis reveals that piecewise linearity imposes
fundamental limitations on the tightness of leading certification techniques.
These limitations are felt in practical terms as a greater need for capacity in
models hoped to be certified efficiently. Moreover, this is in addition to the
capacity necessary to learn a robust boundary, studied in prior work. However,
we argue that addressing the limitations of piecewise linearity through scaling
up model capacity may give rise to potential difficulties -- particularly
regarding robust generalization -- therefore, we conclude by suggesting that
developing smooth activation functions may be the way forward for advancing the
performance of certified neural networks.
