  Deep residual networks (ResNets) have shown state-of-the-art performance in
various real-world applications. Recently, the ResNets model was
reparameterized and interpreted as solutions to a continuous ordinary
differential equation or Neural-ODE model. In this study, we propose a neural
generalized ordinary differential equation (Neural-GODE) model with
layer-varying parameters to further extend the Neural-ODE to approximate the
discrete ResNets. Specifically, we use nonparametric B-spline functions to
parameterize the Neural-GODE so that the trade-off between the model complexity
and computational efficiency can be easily balanced. It is demonstrated that
ResNets and Neural-ODE models are special cases of the proposed Neural-GODE
model. Based on two benchmark datasets, MNIST and CIFAR-10, we show that the
layer-varying Neural-GODE is more flexible and general than the standard
Neural-ODE. Furthermore, the Neural-GODE enjoys the computational and memory
benefits while performing comparably to ResNets in prediction accuracy.
