  How does the input segmentation of pretrained language models (PLMs) affect
their interpretations of complex words? We present the first study
investigating this question, taking BERT as the example PLM and focusing on its
semantic representations of English derivatives. We show that PLMs can be
interpreted as serial dual-route models, i.e., the meanings of complex words
are either stored or else need to be computed from the subwords, which implies
that maximally meaningful input tokens should allow for the best generalization
on new words. This hypothesis is confirmed by a series of semantic probing
tasks on which DelBERT (Derivation leveraging BERT), a model with derivational
input segmentation, substantially outperforms BERT with WordPiece segmentation.
Our results suggest that the generalization capabilities of PLMs could be
further improved if a morphologically-informed vocabulary of input tokens were
used.
