  Neuron analysis provides insights into how knowledge is structured in
representations and discovers the role of neurons in the network. In addition
to developing an understanding of our models, neuron analysis enables various
applications such as debiasing, domain adaptation and architectural search. We
present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis
of natural language processing models. It implements various interpretation
methods under a unified API, and provides a framework for data processing and
evaluation, thus making it easier for researchers and practitioners to perform
neuron analysis. The Python toolkit is available at
https://www.github.com/fdalvi/NeuroX. Demo Video available at
https://youtu.be/mLhs2YMx4u8.
