  This work presents a two-stage framework for progressively developing neural
architectures to adapt/ generalize well on a given training data set. In the
first stage, a manifold-regularized layerwise sparsifying training approach is
adopted where a new layer is added each time and trained independently by
freezing parameters in the previous layers. In order to constrain the functions
that should be learned by each layer, we employ a sparsity regularization term,
manifold regularization term and a physics-informed term. We derive the
necessary conditions for trainability of a newly added layer and analyze the
role of manifold regularization. In the second stage of the Algorithm, a
sequential learning process is adopted where a sequence of small networks is
employed to extract information from the residual produced in stage I and
thereby making robust and more accurate predictions. Numerical investigations
with fully connected network on prototype regression problem, and
classification problem demonstrate that the proposed approach can outperform
adhoc baseline networks. Further, application to physics-informed neural
network problems suggests that the method could be employed for creating
interpretable hidden layers in a deep network while outperforming equivalent
baseline networks.
