  Large language models (LLMs) have achieved impressive performance in numerous
domains but often struggle to process lengthy inputs effectively and
efficiently due to limited length generalization and attention's quadratic
computational demands. Many sought to mitigate this by restricting the
attention window within the pre-trained length. However, these methods
introduce new issues such as ignoring the middle context and requiring
additional training. To address these problems, we propose LongHeads, a
training-free framework that enhances LLM's long context ability by unlocking
multi-head attention's untapped potential. Instead of allowing each head to
attend to the full sentence, which struggles with generalizing to longer
sequences due to out-of-distribution (OOD) issues, we allow each head to
process in-distribution length by selecting and attending to important context
chunks. To this end, we propose a chunk selection strategy that relies on the
inherent correlation between the query and the key representations, efficiently
distributing context chunks to different heads. In this way, each head ensures
it can effectively process attended tokens within the trained length, while
different heads in different layers can collectively process longer contexts.
LongHeads works efficiently in linear time, fits seamlessly with many LLMs that
use relative positional encoding. LongHeads achieves 100% accuracy at the 128k
length on passkey retrieval task, verifying LongHeads's efficacy in extending
the usable context window for existing models. We release our code at
https://github.com/LuLuLuyi/LongHeads .
