  In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long
Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into
the class of polynomial systems. This embedding preserves input-output behavior
and can suitably be extended to other neural DE architectures. We then use
realization theory of polynomial systems to provide necessary conditions for an
input-output map to be realizable by an ODE-LSTM and sufficient conditions for
minimality of such systems. These results represent the first steps towards
realization theory of recurrent neural ODE architectures, which is is expected
be useful for model reduction and learning algorithm analysis of recurrent
neural ODEs.
