  Deep neural networks based on state space models (SSMs) are attracting much
attention in sequence modeling since their computational cost is significantly
smaller than that of Transformers. While the capabilities of SSMs have been
primarily investigated through experimental comparisons, theoretical
understanding of SSMs is still limited. In particular, there is a lack of
statistical and quantitative evaluation of whether SSM can replace
Transformers. In this paper, we theoretically explore in which tasks SSMs can
be alternatives of Transformers from the perspective of estimating
sequence-to-sequence functions. We consider the setting where the target
function has direction-dependent smoothness and prove that SSMs can estimate
such functions with the same convergence rate as Transformers. Additionally, we
prove that SSMs can estimate the target function, even if the smoothness
changes depending on the input sequence, as well as Transformers. Our results
show the possibility that SSMs can replace Transformers when estimating the
functions in certain classes that appear in practice.
