  A proper parametrization of state transition matrices of linear state-space
models (SSMs) followed by standard nonlinearities enables them to efficiently
learn representations from sequential data, establishing the state-of-the-art
on a large series of long-range sequence modeling benchmarks. In this paper, we
show that we can improve further when the structural SSM such as S4 is given by
a linear liquid time-constant (LTC) state-space model. LTC neural networks are
causal continuous-time neural networks with an input-dependent state transition
module, which makes them learn to adapt to incoming inputs at inference. We
show that by using a diagonal plus low-rank decomposition of the state
transition matrix introduced in S4, and a few simplifications, the LTC-based
structural state-space model, dubbed Liquid-S4, achieves the new
state-of-the-art generalization across sequence modeling tasks with long-term
dependencies such as image, text, audio, and medical time-series, with an
average performance of 87.32% on the Long-Range Arena benchmark. On the full
raw Speech Command recognition, dataset Liquid-S4 achieves 96.78% accuracy with
a 30% reduction in parameter counts compared to S4. The additional gain in
performance is the direct result of the Liquid-S4's kernel structure that takes
into account the similarities of the input sequence samples during training and
inference.
