  Recent work has led to the development of an elegant theory of Linearly
Solvable Markov Decision Processes (LMDPs) and related Path-Integral Control
Problems. Traditionally, MDPs have been formulated using stochastic policies
and a control cost based on the KL divergence. In this paper, we extend this
framework to a more general class of divergences: the Renyi divergences. These
are a more general class of divergences parameterized by a continuous parameter
that include the KL divergence as a special case. The resulting control
problems can be interpreted as solving a risk-sensitive version of the LMDP
problem. For a > 0, we get risk-averse behavior (the degree of risk-aversion
increases with a) and for a < 0, we get risk-seeking behavior. We recover LMDPs
in the limit as a -> 0. This work generalizes the recently developed
risk-sensitive path-integral control formalism which can be seen as the
continuous-time limit of results obtained in this paper. To the best of our
knowledge, this is a general theory of linearly solvable control and includes
all previous work as a special case. We also present an alternative
interpretation of these results as solving a 2-player (cooperative or
competitive) Markov Game. From the linearity follow a number of nice properties
including compositionality of control laws and a path-integral representation
of the value function. We demonstrate the usefulness of the framework on
control problems with noise where different values of lead to qualitatively
different control behaviors.
