  The deep neural network suffers from many fundamental issues in machine
learning. For example, it often gets trapped into a local minimum in training,
and its prediction uncertainty is hard to be assessed. To address these issues,
we propose the so-called kernel-expanded stochastic neural network (K-StoNet)
model, which incorporates support vector regression (SVR) as the first hidden
layer and reformulates the neural network as a latent variable model. The
former maps the input vector into an infinite dimensional feature space via a
radial basis function (RBF) kernel, ensuring absence of local minima on its
training loss surface. The latter breaks the high-dimensional nonconvex neural
network training problem into a series of low-dimensional convex optimization
problems, and enables its prediction uncertainty easily assessed. The K-StoNet
can be easily trained using the imputation-regularized optimization (IRO)
algorithm. Compared to traditional deep neural networks, K-StoNet possesses a
theoretical guarantee to asymptotically converge to the global optimum and
enables the prediction uncertainty easily assessed. The performances of the new
model in training, prediction and uncertainty quantification are illustrated by
simulated and real data examples.
