  The advent of Convolutional Neural Networks (CNNs) has led to their
application in several domains. One noteworthy application is the perception
system for autonomous driving that relies on the predictions from CNNs.
Practitioners evaluate the generalization ability of such CNNs by calculating
various metrics on an independent test dataset. A test dataset is often chosen
based on only one precondition, i.e., its elements are not a part of the
training data. Such a dataset may contain objects that are both similar and
novel w.r.t. the training dataset. Nevertheless, existing works do not reckon
the novelty of the test samples and treat them all equally for evaluating
generalization. Such novelty-based evaluations are of significance to validate
the fitness of a CNN in autonomous driving applications. Hence, we propose a
CNN generalization scoring framework that considers novelty of objects in the
test dataset. We begin with the representation learning technique to reduce the
image data into a low-dimensional space. It is on this space we estimate the
novelty of the test samples. Finally, we calculate the generalization score as
a combination of the test data prediction performance and novelty. We perform
an experimental study of the same for our traffic light detection application.
In addition, we systematically visualize the results for an interpretable
notion of novelty.
