  Formal verification of neural networks (NNs) is a challenging and important
problem. Existing efficient complete solvers typically require the
branch-and-bound (BaB) process, which splits the problem domain into
sub-domains and solves each sub-domain using faster but weaker incomplete
verifiers, such as Linear Programming (LP) on linearly relaxed sub-domains. In
this paper, we propose to use the backward mode linear relaxation based
perturbation analysis (LiRPA) to replace LP during the BaB process, which can
be efficiently implemented on the typical machine learning accelerators such as
GPUs and TPUs. However, unlike LP, LiRPA when applied naively can produce much
weaker bounds and even cannot check certain conflicts of sub-domains during
splitting, making the entire procedure incomplete after BaB. To address these
challenges, we apply a fast gradient based bound tightening procedure combined
with batch splits and the design of minimal usage of LP bound procedure,
enabling us to effectively use LiRPA on the accelerator hardware for the
challenging complete NN verification problem and significantly outperform
LP-based approaches. On a single GPU, we demonstrate an order of magnitude
speedup compared to existing LP-based approaches.
