  Pretrained language models (PLMs) form the basis of most state-of-the-art NLP
technologies. Nevertheless, they are essentially black boxes: Humans do not
have a clear understanding of what knowledge is encoded in different parts of
the models, especially in individual neurons. The situation is different in
computer vision, where feature visualization provides a decompositional
interpretability technique for neurons of vision models. Activation
maximization is used to synthesize inherently interpretable visual
representations of the information encoded in individual neurons. Our work is
inspired by this but presents a cautionary tale on the interpretability of
single neurons, based on the first large-scale attempt to adapt activation
maximization to NLP, and, more specifically, large PLMs. We propose feature
textualization, a technique to produce dense representations of neurons in the
PLM word embedding space. We apply feature textualization to the BERT model
(Devlin et al., 2019) to investigate whether the knowledge encoded in
individual neurons can be interpreted and symbolized. We find that the produced
representations can provide insights about the knowledge encoded in individual
neurons, but that individual neurons do not represent clearcut symbolic units
of language such as words. Additionally, we use feature textualization to
investigate how many neurons are needed to encode words in BERT.
