  Recent language models have been improved by the addition of external memory.
Nearest neighbor language models retrieve similar contexts to assist in word
prediction. The addition of locality levels allows a model to learn how to
weight neighbors based on their relative location to the current text in source
documents, and have been shown to further improve model performance. Nearest
neighbor models have been explored for controllable generation but have not
examined the use of locality levels. We present a novel approach for this
purpose and evaluate it using automatic and human evaluation on politeness,
formality, supportiveness, and toxicity textual data. We find that our model is
successfully able to control style and provides a better fluency-style
trade-off than previous work.
