  We establish estimations for the parameters of the output distribution for
the softmax activation function using the probit function. As an application,
we develop a new efficient Bayesian learning algorithm for fully connected
neural networks, where training and predictions are performed within the
Bayesian inference framework in closed-form. This approach allows sequential
learning and requires no computationally expensive gradient calculation and
Monte Carlo sampling. Our work generalizes the Bayesian algorithm for a single
perceptron for binary classification in \cite{H} to multi-layer perceptrons for
multi-class classification.
