  Designing an optimal deep neural network for a given task is important and
challenging in many machine learning applications. To address this issue, we
introduce a self-adaptive algorithm: the adaptive network enhancement (ANE)
method, written as loops of the form train, estimate and enhance. Starting with
a small two-layer neural network (NN), the step train is to solve the
optimization problem at the current NN; the step estimate is to compute a
posteriori estimator/indicators using the solution at the current NN; the step
enhance is to add new neurons to the current NN.
  Novel network enhancement strategies based on the computed
estimator/indicators are developed in this paper to determine how many new
neurons and when a new layer should be added to the current NN. The ANE method
provides a natural process for obtaining a good initialization in training the
current NN; in addition, we introduce an advanced procedure on how to
initialize newly added neurons for a better approximation. We demonstrate that
the ANE method can automatically design a nearly minimal NN for learning
functions exhibiting sharp transitional layers as well as discontinuous
solutions of hyperbolic partial differential equations.
