  The remarkable generalization performance of large-scale models has been
challenging the conventional wisdom of the statistical learning theory.
Although recent theoretical studies have shed light on this behavior in linear
models and nonlinear classifiers, a comprehensive understanding of
overparameterization in nonlinear regression models is still lacking. This
study explores the predictive properties of overparameterized nonlinear
regression within the Bayesian framework, extending the methodology of the
adaptive prior considering the intrinsic spectral structure of the data.
Posterior contraction is established for generalized linear and single-neuron
models with Lipschitz continuous activation functions, demonstrating the
consistency in the predictions of the proposed approach. Moreover, the Bayesian
framework enables uncertainty estimation of the predictions. The proposed
method was validated via numerical simulations and a real data application,
showing its ability to achieve accurate predictions and reliable uncertainty
estimates. This work provides a theoretical understanding of the advantages of
overparameterization and a principled Bayesian approach to large nonlinear
models.
