  In this paper, we investigate the length-extension of state-space models
(SSMs) in language modeling. Length extension involves training models on short
sequences and testing them on longer ones. We show that state-space models
trained with zero hidden states initialization have difficulty doing length
extension. We explain this difficulty by pointing out the length extension is
equivalent to polynomial extrapolation. Based on the theory, we propose a
simple yet effective method - changing the hidden states initialization scheme
- to improve the length extension. Moreover, our method shows that using long
training sequence length is beneficial but not necessary to length extension.
Changing the hidden state initialization enables the efficient training of
long-memory model with a smaller training context length.
