  Bayesian neural networks (BNNs) have been an important framework in the study
of uncertainty quantification. Deterministic variational inference, one of the
inference methods, utilizes moment propagation to compute the predictive
distributions and objective functions. Unfortunately, deriving the moments
requires computationally expensive Taylor expansion in nonlinear functions,
such as a rectified linear unit (ReLU) or a sigmoid function. Therefore, a new
nonlinear function that realizes faster moment propagation than conventional
functions is required. In this paper, we propose a novel nonlinear function
named moment propagating-Gaussian error linear unit (MP-GELU) that enables the
fast derivation of first and second moments in BNNs. MP-GELU enables the
analytical computation of moments by applying nonlinearity to the input
statistics, thereby reducing the computationally expensive calculations
required for nonlinear functions. In empirical experiments on regression tasks,
we observed that the proposed MP-GELU provides higher prediction accuracy and
better quality of uncertainty with faster execution than those of ReLU-based
BNNs.
