  Vocabulary tests, once a cornerstone of language modeling evaluation, have
been largely overlooked in the current landscape of Large Language Models
(LLMs) like Llama, Mistral, and GPT. While most LLM evaluation benchmarks focus
on specific tasks or domain-specific knowledge, they often neglect the
fundamental linguistic aspects of language understanding and production. In
this paper, we advocate for the revival of vocabulary tests as a valuable tool
for assessing LLM performance. We evaluate seven LLMs using two vocabulary test
formats across two languages and uncover surprising gaps in their lexical
knowledge. These findings shed light on the intricacies of LLM word
representations, their learning mechanisms, and performance variations across
models and languages. Moreover, the ability to automatically generate and
perform vocabulary tests offers new opportunities to expand the approach and
provide a more complete picture of LLMs' language skills.
