  The paper deals with learning probability distributions of observed data by
artificial neural networks. We suggest a so-called gradient conjugate prior
(GCP) update appropriate for neural networks, which is a modification of the
classical Bayesian update for conjugate priors. We establish a connection
between the gradient conjugate prior update and the maximization of the
log-likelihood of the predictive distribution. Unlike for the Bayesian neural
networks, we use deterministic weights of neural networks, but rather assume
that the ground truth distribution is normal with unknown mean and variance and
learn by the neural networks the parameters of a prior (normal-gamma
distribution) for these unknown mean and variance. The update of the parameters
is done, using the gradient that, at each step, directs towards minimizing the
Kullback--Leibler divergence from the prior to the posterior distribution (both
being normal-gamma). We obtain a corresponding dynamical system for the prior's
parameters and analyze its properties. In particular, we study the limiting
behavior of all the prior's parameters and show how it differs from the case of
the classical full Bayesian update. The results are validated on synthetic and
real world data sets.
