  Deep neural networks (DNNs) are widely used in various application domains
such as image processing, speech recognition, and natural language processing.
However, testing DNN models may be challenging due to the complexity and size
of their input domain. Particularly, testing DNN models often requires
generating or exploring large unlabeled datasets. In practice, DNN test
oracles, which identify the correct outputs for inputs, often require expensive
manual effort to label test data, possibly involving multiple experts to ensure
labeling correctness. In this paper, we propose DeepGD, a black-box
multi-objective test selection approach for DNN models. It reduces the cost of
labeling by prioritizing the selection of test inputs with high fault revealing
power from large unlabeled datasets. DeepGD not only selects test inputs with
high uncertainty scores to trigger as many mispredicted inputs as possible but
also maximizes the probability of revealing distinct faults in the DNN model by
selecting diverse mispredicted inputs. The experimental results conducted on
four widely used datasets and five DNN models show that in terms of
fault-revealing ability: (1) White-box, coverage-based approaches fare poorly,
(2) DeepGD outperforms existing black-box test selection approaches in terms of
fault detection, and (3) DeepGD also leads to better guidance for DNN model
retraining when using selected inputs to augment the training set.
