  The challenge of assigning importance to individual neurons in a network is
of interest when interpreting deep learning models. In recent work, Dhamdhere
et al. proposed Total Conductance, a "natural refinement of Integrated
Gradients" for attributing importance to internal neurons. Unfortunately, the
authors found that calculating conductance in tensorflow required the addition
of several custom gradient operators and did not scale well. In this work, we
show that the formula for Total Conductance is mathematically equivalent to
Path Integrated Gradients computed on a hidden layer in the network. We provide
a scalable implementation of Total Conductance using standard tensorflow
gradient operators that we call Neuron Integrated Gradients. We compare Neuron
Integrated Gradients to DeepLIFT, a pre-existing computationally efficient
approach that is applicable to calculating internal neuron importance. We find
that DeepLIFT produces strong empirical results and is faster to compute, but
because it lacks the theoretical properties of Neuron Integrated Gradients, it
may not always be preferred in practice. Colab notebook reproducing results:
http://bit.ly/neuronintegratedgradients
