  Code-trained language models have proven to be highly effective for various
code intelligence tasks. However, they can be challenging to train and deploy
for many software engineering applications due to computational bottlenecks and
memory constraints. Implementing effective strategies to address these issues
requires a better understanding of these 'black box' models. In this paper, we
perform the first neuron-level analysis for source code models to identify
\textit{important} neurons within latent representations. We achieve this by
eliminating neurons that are highly similar or irrelevant to the given task.
This approach helps us understand which neurons and layers can be eliminated
(redundancy analysis) and where important code properties are located within
the network (concept analysis). Using redundancy analysis, we make observations
relevant to knowledge transfer and model optimization applications. We find
that over 95\% of the neurons are redundant with respect to our code
intelligence tasks and can be eliminated without significant loss in accuracy.
We also discover several subsets of neurons that can make predictions with
baseline accuracy. Through concept analysis, we explore the traceability and
distribution of human-recognizable concepts within latent code representations
which could be used to influence model predictions. We trace individual and
subsets of important neurons to specific code properties and identify 'number'
neurons, 'string' neurons, and higher-level 'text' neurons for token-level
tasks and higher-level concepts important for sentence-level downstream tasks.
This also helps us understand how decomposable and transferable task-related
features are and can help devise better techniques for transfer learning, model
compression, and the decomposition of deep neural networks into modules.
