  The current deep learning model is of a single-grade, that is, it learns a
deep neural network by solving a single nonconvex optimization problem. When
the layer number of the neural network is large, it is computationally
challenging to carry out such a task efficiently. Inspired by the human
education process which arranges learning in grades, we propose a multi-grade
learning model: We successively solve a number of optimization problems of
small sizes, which are organized in grades, to learn a shallow neural network
for each grade. Specifically, the current grade is to learn the leftover from
the previous grade. In each of the grades, we learn a shallow neural network
stacked on the top of the neural network, learned in the previous grades, which
remains unchanged in training of the current and future grades. By dividing the
task of learning a deep neural network into learning several shallow neural
networks, one can alleviate the severity of the nonconvexity of the original
optimization problem of a large size. When all grades of the learning are
completed, the final neural network learned is a stair-shape neural network,
which is the superposition of networks learned from all grades. Such a model
enables us to learn a deep neural network much more effectively and
efficiently. Moreover, multi-grade learning naturally leads to adaptive
learning. We prove that in the context of function approximation if the neural
network generated by a new grade is nontrivial, the optimal error of the grade
is strictly reduced from the optimal error of the previous grade. Furthermore,
we provide several proof-of-concept numerical examples which demonstrate that
the proposed multi-grade model outperforms significantly the traditional
single-grade model and is much more robust than the traditional model.
