  In this work we study the presence of expert units in pre-trained Transformer
Models (TM), and how they impact a model's performance. We define expert units
to be neurons that are able to classify a concept with a given average
precision, where a concept is represented by a binary set of sentences
containing the concept (or not). Leveraging the OneSec dataset (Scarlini et
al., 2019), we compile a dataset of 1641 concepts that allows diverse expert
units in TM to be discovered. We show that expert units are important in
several ways: (1) The presence of expert units is correlated ($r^2=0.833$) with
the generalization power of TM, which allows ranking TM without requiring
fine-tuning on suites of downstream tasks. We further propose an empirical
method to decide how accurate such experts should be to evaluate
generalization. (2) The overlap of top experts between concepts provides a
sensible way to quantify concept co-learning, which can be used for
explainability of unknown concepts. (3) We show how to self-condition
off-the-shelf pre-trained language models to generate text with a given concept
by forcing the top experts to be active, without requiring re-training the
model or using additional parameters.
