  While successful in many fields, deep neural networks (DNNs) still suffer
from some open problems such as bad local minima and unsatisfactory
generalization performance. In this work, we propose a novel architecture
called Maximum-and-Concatenation Networks (MCN) to try eliminating bad local
minima and improving generalization ability as well. Remarkably, we prove that
MCN has a very nice property; that is, \emph{every local minimum of an
$(l+1)$-layer MCN can be better than, at least as good as, the global minima of
the network consisting of its first $l$ layers}. In other words, by increasing
the network depth, MCN can autonomously improve its local minima's goodness,
what is more, \emph{it is easy to plug MCN into an existing deep model to make
it also have this property}. Finally, under mild conditions, we show that MCN
can approximate certain continuous functions arbitrarily well with \emph{high
efficiency}; that is, the covering number of MCN is much smaller than most
existing DNNs such as deep ReLU. Based on this, we further provide a tight
generalization bound to guarantee the inference ability of MCN when dealing
with testing samples.
