  What are the functionals of the reward that can be computed and optimized
exactly in Markov Decision Processes?In the finite-horizon, undiscounted
setting, Dynamic Programming (DP) can only handle these operations efficiently
for certain classes of statistics. We summarize the characterization of these
classes for policy evaluation, and give a new answer for the planning problem.
Interestingly, we prove that only generalized means can be optimized exactly,
even in the more general framework of Distributional Reinforcement Learning
(DistRL).DistRL permits, however, to evaluate other functionals approximately.
We provide error bounds on the resulting estimators, and discuss the potential
of this approach as well as its limitations.These results contribute to
advancing the theory of Markov Decision Processes by examining overall
characteristics of the return, and particularly risk-conscious strategies.
