  Discovering a suitable neural network architecture for modeling complex
dynamical systems poses a formidable challenge, often involving extensive trial
and error and navigation through a high-dimensional hyper-parameter space. In
this paper, we discuss a systematic approach to constructing neural
architectures for modeling a subclass of dynamical systems, namely, Linear
Time-Invariant (LTI) systems. We use a variant of continuous-time neural
networks in which the output of each neuron evolves continuously as a solution
of a first-order or second-order Ordinary Differential Equation (ODE). Instead
of deriving the network architecture and parameters from data, we propose a
gradient-free algorithm to compute sparse architecture and network parameters
directly from the given LTI system, leveraging its properties. We bring forth a
novel neural architecture paradigm featuring horizontal hidden layers and
provide insights into why employing conventional neural architectures with
vertical hidden layers may not be favorable. We also provide an upper bound on
the numerical errors of our neural networks. Finally, we demonstrate the high
accuracy of our constructed networks on three numerical examples.
