  The impressive performance of Convolutional Neural Networks (CNNs) when
solving different vision problems is shadowed by their black-box nature and our
consequent lack of understanding of the representations they build and how
these representations are organized. To help understanding these issues, we
propose to describe the activity of individual neurons by their Neuron Feature
visualization and quantify their inherent selectivity with two specific
properties. We explore selectivity indexes for: an image feature (color); and
an image label (class membership). Our contribution is a framework to seek or
classify neurons by indexing on these selectivity properties. It helps to find
color selective neurons, such as a red-mushroom neuron in layer Conv4 or class
selective neurons such as dog-face neurons in layer Conv5 in VGG-M, and
establishes a methodology to derive other selectivity properties. Indexing on
neuron selectivity can statistically draw how features and classes are
represented through layers in a moment when the size of trained nets is growing
and automatic tools to index neurons can be helpful.
