  Reinforcement learning (RL) allows an agent interacting sequentially with an
environment to maximize its long-term expected return. In the distributional RL
(DistrRL) paradigm, the agent goes beyond the limit of the expected value, to
capture the underlying probability distribution of the return across all time
steps. The set of DistrRL algorithms has led to improved empirical performance.
Nevertheless, the theory of DistrRL is still not fully understood, especially
in the control case. In this paper, we present the simpler one-step
distributional reinforcement learning (OS-DistrRL) framework encompassing only
the randomness induced by the one-step dynamics of the environment. Contrary to
DistrRL, we show that our approach comes with a unified theory for both policy
evaluation and control. Indeed, we propose two OS-DistrRL algorithms for which
we provide an almost sure convergence analysis. The proposed approach compares
favorably with categorical DistrRL on various environments.
