  Long text generation, such as novel writing and discourse-level translation
with extremely long contexts, presents significant challenges to current
language models. Existing methods mainly focus on extending the model's context
window through strategies like length extrapolation. However, these approaches
demand substantial hardware resources during the training and/or inference
phases. Our proposed method, Temp-Lora, introduces an alternative concept.
Instead of relying on the KV cache to store all context information, we embeds
this information directly into a temporary Lora module. In the process of long
text generation, this module is progressively trained with text generated
previously. This approach not only efficiently preserves contextual knowledge
but also prevents any permanent alteration to the model's parameters given that
the module is discarded post-generation. Extensive experiments on the PG19
language modeling benchmark and the GuoFeng discourse-level translation
benchmark validate the effectiveness of Temp-Lora. Our results show that: 1)
Temp-Lora substantially enhances generation quality for long text, as indicated
by a 13.2% decrease in perplexity (PPL) on a subset of PG19, and a 29.3%
decrease in PPL along with a 113.2% increase in BLEU score on a subset of
GuoFeng, 2) Temp-Lora is compatible with and enhances most existing long text
generation methods, and 3) Temp-Lora can greatly reduce computational costs by
shortening the context window. For example, we can ensure a moderate
improvement in generation quality (a decrease of 3.8% in PPL) while enabling a
51.5% memory usage reduction and a 60.0% decrease in latency for inference.
