  Continuous-depth neural networks, such as the Neural Ordinary Differential
Equations (ODEs), have aroused a great deal of interest from the communities of
machine learning and data science in recent years, which bridge the connection
between deep neural networks and dynamical systems. In this article, we
introduce a new sort of continuous-depth neural network, called the Neural
Piecewise-Constant Delay Differential Equations (PCDDEs). Here, unlike the
recently proposed framework of the Neural Delay Differential Equations (DDEs),
we transform the single delay into the piecewise-constant delay(s). The Neural
PCDDEs with such a transformation, on one hand, inherit the strength of
universal approximating capability in Neural DDEs. On the other hand, the
Neural PCDDEs, leveraging the contributions of the information from the
multiple previous time steps, further promote the modeling capability without
augmenting the network dimension. With such a promotion, we show that the
Neural PCDDEs do outperform the several existing continuous-depth neural
frameworks on the one-dimensional piecewise-constant delay population dynamics
and real-world datasets, including MNIST, CIFAR10, and SVHN.
