  Interpretability of Deep Neural Networks has become a major area of
exploration. Although these networks have achieved state of the art accuracy in
many tasks, it is extremely difficult to interpret and explain their decisions.
In this work we analyze the final and penultimate layers of Deep Convolutional
Networks and provide an efficient method for identifying subsets of features
that contribute most towards the network's decision for a class. We demonstrate
that the number of such features per class is much lower in comparison to the
dimension of the final layer and therefore the decision surface of Deep CNNs
lies on a low dimensional manifold and is proportional to the network depth.
Our methods allow to decompose the final layer into separate subspaces which is
far more interpretable and has a lower computational cost as compared to the
final layer of the full network.
