  Robust Markov Decision Processes (RMDPs) have received significant research
interest, offering an alternative to standard Markov Decision Processes (MDPs)
that often assume fixed transition probabilities. RMDPs address this by
optimizing for the worst-case scenarios within ambiguity sets. While earlier
studies on RMDPs have largely centered on risk-neutral reinforcement learning
(RL), with the goal of minimizing expected total discounted costs, in this
paper, we analyze the robustness of CVaR-based risk-sensitive RL under RMDP.
Firstly, we consider predetermined ambiguity sets. Based on the coherency of
CVaR, we establish a connection between robustness and risk sensitivity, thus,
techniques in risk-sensitive RL can be adopted to solve the proposed problem.
Furthermore, motivated by the existence of decision-dependent uncertainty in
real-world problems, we study problems with state-action-dependent ambiguity
sets. To solve this, we define a new risk measure named NCVaR and build the
equivalence of NCVaR optimization and robust CVaR optimization. We further
propose value iteration algorithms and validate our approach in simulation
experiments.
