  The proliferation of deep neural networks in various domains has seen an
increased need for interpretability of these models. Preliminary work done
along this line and papers that surveyed such, are focused on high-level
representation analysis. However, a recent branch of work has concentrated on
interpretability at a more granular level of analyzing neurons within these
models. In this paper, we survey the work done on neuron analysis including: i)
methods to discover and understand neurons in a network, ii) evaluation
methods, iii) major findings including cross architectural comparisons that
neuron analysis has unraveled, iv) applications of neuron probing such as:
controlling the model, domain adaptation etc., and v) a discussion on open
issues and future research directions.
