  Transformer-based large language models (LLMs) typically have a limited
context window, resulting in significant performance degradation when
processing text beyond the length of the context window. Extensive studies have
been proposed to extend the context window and achieve length extrapolation of
LLMs, but there is still a lack of in-depth interpretation of these approaches.
In this study, we explore the positional information within and beyond the
context window for deciphering the underlying mechanism of LLMs. By using a
mean-based decomposition method, we disentangle positional vectors from hidden
states of LLMs and analyze their formation and effect on attention.
Furthermore, when texts exceed the context window, we analyze the change of
positional vectors in two settings, i.e., direct extrapolation and context
window extension. Based on our findings, we design two training-free context
window extension methods, positional vector replacement and attention window
extension. Experimental results show that our methods can effectively extend
the context window length.
