  Motivated by the need for a robust policy in the face of environment shifts
between training and the deployment, we contribute to the theoretical
foundation of distributionally robust reinforcement learning (DRRL). This is
accomplished through a comprehensive modeling framework centered around
distributionally robust Markov decision processes (DRMDPs). This framework
obliges the decision maker to choose an optimal policy under the worst-case
distributional shift orchestrated by an adversary. By unifying and extending
existing formulations, we rigorously construct DRMDPs that embraces various
modeling attributes for both the decision maker and the adversary. These
attributes include adaptability granularity, exploring history-dependent,
Markov, and Markov time-homogeneous decision maker and adversary dynamics.
Additionally, we delve into the flexibility of shifts induced by the adversary,
examining SA and S-rectangularity. Within this DRMDP framework, we investigate
conditions for the existence or absence of the dynamic programming principle
(DPP). From an algorithmic standpoint, the existence of DPP holds significant
implications, as the vast majority of existing data and computationally
efficiency RL algorithms are reliant on the DPP. To study its existence, we
comprehensively examine combinations of controller and adversary attributes,
providing streamlined proofs grounded in a unified methodology. We also offer
counterexamples for settings in which a DPP with full generality is absent.
