  Learning meaningful word embeddings is key to training a robust language
model. The recent rise of Large Language Models (LLMs) has provided us with
many new word/sentence/document embedding models. Although LLMs have shown
remarkable advancement in various NLP tasks, it is still unclear whether the
performance improvement is merely because of scale or whether underlying
embeddings they produce significantly differ from classical encoding models
like Sentence-BERT (SBERT) or Universal Sentence Encoder (USE). This paper
systematically investigates this issue by comparing classical word embedding
techniques against LLM-based word embeddings in terms of their latent vector
semantics. Our results show that LLMs tend to cluster semantically related
words more tightly than classical models. LLMs also yield higher average
accuracy on the Bigger Analogy Test Set (BATS) over classical methods. Finally,
some LLMs tend to produce word embeddings similar to SBERT, a relatively
lighter classical model.
