  Neural Ordinary Differential Equations (NODEs), a framework of
continuous-depth neural networks, have been widely applied, showing exceptional
efficacy in coping with representative datasets. Recently, an augmented
framework has been developed to overcome some limitations that emerged in the
application of the original framework. In this paper, we propose a new class of
continuous-depth neural networks with delay, named Neural Delay Differential
Equations (NDDEs). To compute the corresponding gradients, we use the adjoint
sensitivity method to obtain the delayed dynamics of the adjoint. Differential
equations with delays are typically seen as dynamical systems of infinite
dimension that possess more fruitful dynamics. Compared to NODEs, NDDEs have a
stronger capacity of nonlinear representations. We use several illustrative
examples to demonstrate this outstanding capacity. Firstly, we successfully
model the delayed dynamics where the trajectories in the lower-dimensional
phase space could be mutually intersected and even chaotic in a model-free or
model-based manner. Traditional NODEs, without any argumentation, are not
directly applicable for such modeling. Secondly, we achieve lower loss and
higher accuracy not only for the data produced synthetically by complex models
but also for the CIFAR10, a well-known image dataset. Our results on the NDDEs
demonstrate that appropriately articulating the elements of dynamical systems
into the network design is truly beneficial in promoting network performance.
