  Distributional reinforcement learning (DRL) enhances the understanding of the
effects of the randomness in the environment by letting agents learn the
distribution of a random return, rather than its expected value as in standard
RL. At the same time, a main challenge in DRL is that policy evaluation in DRL
typically relies on the representation of the return distribution, which needs
to be carefully designed. In this paper, we address this challenge for a
special class of DRL problems that rely on linear quadratic regulator (LQR) for
control, advocating for a new distributional approach to LQR, which we call
\emph{distributional LQR}. Specifically, we provide a closed-form expression of
the distribution of the random return which, remarkably, is applicable to all
exogenous disturbances on the dynamics, as long as they are independent and
identically distributed (i.i.d.). While the proposed exact return distribution
consists of infinitely many random variables, we show that this distribution
can be approximated by a finite number of random variables, and the associated
approximation error can be analytically bounded under mild assumptions. Using
the approximate return distribution, we propose a zeroth-order policy gradient
algorithm for risk-averse LQR using the Conditional Value at Risk (CVaR) as a
measure of risk. Numerical experiments are provided to illustrate our
theoretical results.
