  Recently, there has been a significant growth of interest in applying
software engineering techniques for the quality assurance of deep learning (DL)
systems. One popular direction is deep learning testing, where adversarial
examples (a.k.a.~bugs) of DL systems are found either by fuzzing or guided
search with the help of certain testing metrics. However, recent studies have
revealed that the commonly used neuron coverage metrics by existing DL testing
approaches are not correlated to model robustness. It is also not an effective
measurement on the confidence of the model robustness after testing. In this
work, we address this gap by proposing a novel testing framework called
Robustness-Oriented Testing (RobOT). A key part of RobOT is a quantitative
measurement on 1) the value of each test case in improving model robustness
(often via retraining), and 2) the convergence quality of the model robustness
improvement. RobOT utilizes the proposed metric to automatically generate test
cases valuable for improving model robustness. The proposed metric is also a
strong indicator on how well robustness improvement has converged through
testing. Experiments on multiple benchmark datasets confirm the effectiveness
and efficiency of RobOT in improving DL model robustness, with 67.02% increase
on the adversarial robustness that is 50.65% higher than the state-of-the-art
work DeepGini.
