  Recent works have shown that interval bound propagation (IBP) can be used to
train verifiably robust neural networks. Reseachers observe an intriguing
phenomenon on these IBP trained networks: CROWN, a bounding method based on
tight linear relaxation, often gives very loose bounds on these networks. We
also observe that most neurons become dead during the IBP training process,
which could hurt the representation capability of the network. In this paper,
we study the relationship between IBP and CROWN, and prove that CROWN is always
tighter than IBP when choosing appropriate bounding lines. We further propose a
relaxed version of CROWN, linear bound propagation (LBP), that can be used to
verify large networks to obtain lower verified errors than IBP. We also design
a new activation function, parameterized ramp function (ParamRamp), which has
more diversity of neuron status than ReLU. We conduct extensive experiments on
MNIST, CIFAR-10 and Tiny-ImageNet with ParamRamp activation and achieve
state-of-the-art verified robustness. Code and the appendix are available at
https://github.com/ZhaoyangLyu/VerifiablyRobustNN.
