  Bayesian neural networks utilize probabilistic layers that capture
uncertainty over weights and activations, and are trained using Bayesian
inference. Since these probabilistic layers are designed to be drop-in
replacement of their deterministic counter parts, Bayesian neural networks
provide a direct and natural way to extend conventional deep neural networks to
support probabilistic deep learning. However, it is nontrivial to understand,
design and train Bayesian neural networks due to their complexities. We discuss
the essentials of Bayesian neural networks including duality (deep neural
networks, probabilistic models), approximate Bayesian inference, Bayesian
priors, Bayesian posteriors, and deep variational learning. We use TensorFlow
Probability APIs and code examples for illustration. The main problem with
Bayesian neural networks is that the architecture of deep neural networks makes
it quite redundant, and costly, to account for uncertainty for a large number
of successive layers. Hybrid Bayesian neural networks, which use few
probabilistic layers judicially positioned in the networks, provide a practical
solution.
