  Language models (LMs) can generate hallucinations and incoherent outputs,
which highlights their weak context dependency. Cache-LMs, which augment LMs
with a memory of recent history, can increase context dependency and have shown
remarkable performance in diverse language generation tasks. However, we find
that even with training, the performance gain stemming from the cache component
of current cache-LMs is suboptimal due to the misalignment between the current
hidden states and those stored in the memory. In this work, we present
HistAlign, a new training approach to ensure good cache alignment such that the
model receives useful signals from the history. We first prove our concept on a
simple and synthetic task where the memory is essential for correct
predictions, and we show that the cache component of HistAlign is better
aligned and improves overall performance. Next, we evaluate HistAlign on
diverse downstream language generation tasks, including prompt continuation,
abstractive summarization, and data-to-text. We demonstrate that HistAlign
improves text coherence and faithfulness in open-ended and conditional
generation settings respectively. HistAlign is also generalizable across
different model families, showcasing its strength in improving context
dependency of LMs in diverse scenarios. Our code is publicly available at
https://github.com/meetdavidwan/histalign
