  In this paper, after analyzing the reasons of poor generalization and
overfitting in neural networks, we consider some noise data as a singular value
of a continuous function - jump discontinuity point. The continuous part can be
approximated with the simplest neural networks, which have good generalization
performance and optimal network architecture, by traditional algorithms such as
constructive algorithm for feed-forward neural networks with incremental
training, BP algorithm, ELM algorithm, various constructive algorithm, RBF
approximation and SVM. At the same time, we will construct RBF neural networks
to fit the singular value with every error in, and we prove that a function
with jumping discontinuity points can be approximated by the simplest neural
networks with a decay RBF neural networks in by each error, and a function with
jumping discontinuity point can be constructively approximated by a decay RBF
neural networks in by each error and the constructive part have no
generalization influence to the whole machine learning system which will
optimize neural network architecture and generalization performance, reduce the
overfitting phenomenon by avoid fitting the noisy data.
