  Do pretrained language models have knowledge regarding the surface
information of tokens? We examined the surface information stored in word or
subword embeddings acquired by pretrained language models from the perspectives
of token length, substrings, and token constitution. Additionally, we evaluated
the ability of models to generate knowledge regarding token surfaces. We
focused on 12 pretrained language models that were mainly trained on English
and Japanese corpora. Experimental results demonstrate that pretrained language
models have knowledge regarding token length and substrings but not token
constitution. Additionally, the results imply that there is a bottleneck on the
decoder side in terms of effectively utilizing acquired knowledge.
