  We theoretically study the landscape of the training error for neural
networks in overparameterized cases. We consider three basic methods for
embedding a network into a wider one with more hidden units, and discuss
whether a minimum point of the narrower network gives a minimum or saddle point
of the wider one. Our results show that the networks with smooth and ReLU
activation have different partially flat landscapes around the embedded point.
We also relate these results to a difference of their generalization abilities
in overparameterized realization.
