  As robustness verification methods are becoming more precise, training
certifiably robust neural networks is becoming ever more relevant. To this end,
certified training methods compute and then optimize an upper bound on the
worst-case loss over a robustness specification. Curiously, training methods
based on the imprecise interval bound propagation (IBP) consistently outperform
those leveraging more precise bounding methods. Still, we lack an understanding
of the mechanisms making IBP so successful.
  In this work, we thoroughly investigate these mechanisms by leveraging a
novel metric measuring the tightness of IBP bounds. We first show theoretically
that, for deep linear models, tightness decreases with width and depth at
initialization, but improves with IBP training, given sufficient network width.
We, then, derive sufficient and necessary conditions on weight matrices for IBP
bounds to become exact and demonstrate that these impose strong regularization,
explaining the empirically observed trade-off between robustness and accuracy
in certified training.
  Our extensive experimental evaluation validates our theoretical predictions
for ReLU networks, including that wider networks improve performance, yielding
state-of-the-art results. Interestingly, we observe that while all IBP-based
training methods lead to high tightness, this is neither sufficient nor
necessary to achieve high certifiable robustness. This hints at the existence
of new training methods that do not induce the strong regularization required
for tight IBP bounds, leading to improved robustness and standard accuracy.
