  Language models (LMs) automatically learn word embeddings during pre-training
on language corpora. Although word embeddings are usually interpreted as
feature vectors for individual words, their roles in language model generation
remain underexplored. In this work, we theoretically and empirically revisit
output word embeddings and find that their linear transformations are
equivalent to steering language model generation styles. We name such steers
LM-Steers and find them existing in LMs of all sizes. It requires learning
parameters equal to 0.2% of the original LMs' size for steering each style. On
tasks such as language model detoxification and sentiment control, LM-Steers
can achieve comparable or superior performance compared with state-of-the-art
controlled generation methods while maintaining a better balance with
generation quality. The learned LM-Steer serves as a lens in text styles: it
reveals that word embeddings are interpretable when associated with language
model generations and can highlight text spans that most indicate the style
differences. An LM-Steer is transferrable between different language models by
an explicit form calculation. One can also continuously steer LMs simply by
scaling the LM-Steer or compose multiple LM-Steers by adding their
transformations. Our codes are publicly available at
\url{https://github.com/Glaciohound/LM-Steer}.
