  In distributional reinforcement learning not only expected returns but the
complete return distributions of a policy are taken into account. The return
distribution for a fixed policy is given as the solution of an associated
distributional Bellman equation. In this note we consider general
distributional Bellman equations and study existence and uniqueness of their
solutions as well as tail properties of return distributions. We give necessary
and sufficient conditions for existence and uniqueness of return distributions
and identify cases of regular variation. We link distributional Bellman
equations to multivariate affine distributional equations. We show that any
solution of a distributional Bellman equation can be obtained as the vector of
marginal laws of a solution to a multivariate affine distributional equation.
This makes the general theory of such equations applicable to the
distributional reinforcement learning setting.
