  In reinforcement learning, the objective is almost always defined as a
\emph{cumulative} function over the rewards along the process. However, there
are many optimal control and reinforcement learning problems in various
application fields, especially in communications and networking, where the
objectives are not naturally expressed as summations of the rewards. In this
paper, we recognize the prevalence of non-cumulative objectives in various
problems, and propose a modification to existing algorithms for optimizing such
objectives. Specifically, we dive into the fundamental building block for many
optimal control and reinforcement learning algorithms: the Bellman optimality
equation. To optimize a non-cumulative objective, we replace the original
summation operation in the Bellman update rule with a generalized operation
corresponding to the objective. Furthermore, we provide sufficient conditions
on the form of the generalized operation as well as assumptions on the Markov
decision process under which the globally optimal convergence of the
generalized Bellman updates can be guaranteed. We demonstrate the idea
experimentally with the bottleneck objective, i.e., the objectives determined
by the minimum reward along the process, on classical optimal control and
reinforcement learning tasks, as well as on two network routing problems on
maximizing the flow rates.
