  In this paper, we build a framework that facilitates the analysis of
discounted infinite horizon Markov Decision Processes (MDPs) by visualizing
them as deterministic processes where the states are probability measures on
the original state space and the actions are stochastic kernels on the original
action space. We provide a simple general algebraic approach to lifting any MDP
to this space of measures; we call this to measurize the original stochastic
MDP. We show that measurized MDPs are in fact a generalization of stochastic
MDPs, thus the measurized framework can be deployed without loss of fidelity.
Lifting an MDP can be convenient because the measurized framework enables
constraints and value function approximations that are not easily available
from the standard MDP setting. For instance, one can add restrictions or build
approximations based on moments, quantiles, risk measures, etc. Moreover, since
the measurized counterpart to any MDP is deterministic, the measurized
optimality equations trade the complexity of dealing with the expected value
function that appears in the stochastic optimality equations with a more
complex state space.
