  We use a dataset of U.S. first names with labels based on predominant gender
and racial group to examine the effect of training corpus frequency on
tokenization, contextualization, similarity to initial representation, and bias
in BERT, GPT-2, T5, and XLNet. We show that predominantly female and non-white
names are less frequent in the training corpora of these four language models.
We find that infrequent names are more self-similar across contexts, with
Spearman's r between frequency and self-similarity as low as -.763. Infrequent
names are also less similar to initial representation, with Spearman's r
between frequency and linear centered kernel alignment (CKA) similarity to
initial representation as high as .702. Moreover, we find Spearman's r between
racial bias and name frequency in BERT of .492, indicating that lower-frequency
minority group names are more associated with unpleasantness. Representations
of infrequent names undergo more processing, but are more self-similar,
indicating that models rely on less context-informed representations of
uncommon and minority names which are overfit to a lower number of observed
contexts.
