  Despite many algorithmic advances, our theoretical understanding of practical
distributional reinforcement learning methods remains limited. One exception is
Rowland et al. (2018)'s analysis of the C51 algorithm in terms of the Cram\'er
distance, but their results only apply to the tabular setting and ignore C51's
use of a softmax to produce normalized distributions. In this paper we adapt
the Cram\'er distance to deal with arbitrary vectors. From it we derive a new
distributional algorithm which is fully Cram\'er-based and can be combined to
linear function approximation, with formal guarantees in the context of policy
evaluation. In allowing the model's prediction to be any real vector, we lose
the probabilistic interpretation behind the method, but otherwise maintain the
appealing properties of distributional approaches. To the best of our
knowledge, ours is the first proof of convergence of a distributional algorithm
combined with function approximation. Perhaps surprisingly, our results provide
evidence that Cram\'er-based distributional methods may perform worse than
directly approximating the value function.
