  Prior work on safe Reinforcement Learning (RL) has studied risk-aversion to
randomness in dynamics (aleatory) and to model uncertainty (epistemic) in
isolation. We propose and analyze a new framework to jointly model the risk
associated with epistemic and aleatory uncertainties in finite-horizon and
discounted infinite-horizon MDPs. We call this framework that combines
Risk-Averse and Soft-Robust methods RASR. We show that when the risk-aversion
is defined using either EVaR or the entropic risk, the optimal policy in RASR
can be computed efficiently using a new dynamic program formulation with a
time-dependent risk level. As a result, the optimal risk-averse policies are
deterministic but time-dependent, even in the infinite-horizon discounted
setting. We also show that particular RASR objectives reduce to risk-averse RL
with mean posterior transition probabilities. Our empirical results show that
our new algorithms consistently mitigate uncertainty as measured by EVaR and
other standard risk measures.
