  In recent years convolutional neural networks (CNN) have shown striking
progress in various tasks. However, despite the high performance, the training
and prediction process remains to be a black box, leaving it a mystery to
extract what neurons learn in CNN. In this paper, we address the problem of
interpreting a CNN from the aspects of the input image's focus and preference,
and the neurons' domination, activation and contribution to a concrete final
prediction. Specifically, we use two techniques - visualization and clustering
- to tackle the problems above. Visualization means the method of gradient
descent on image pixel, and in clustering section two algorithms are proposed
to cluster respectively over image categories and network neurons. Experiments
and quantitative analyses have demonstrated the effectiveness of the two
methods in explaining the question: what do neurons learn.
