  Offline reinforcement learning (RL) is suitable for safety-critical domains
where online exploration is too costly or dangerous. In such safety-critical
settings, decision-making should take into consideration the risk of
catastrophic outcomes. In other words, decision-making should be
risk-sensitive. Previous works on risk in offline RL combine together offline
RL techniques, to avoid distributional shift, with risk-sensitive RL
algorithms, to achieve risk-sensitivity. In this work, we propose
risk-sensitivity as a mechanism to jointly address both of these issues. Our
model-based approach is risk-averse to both epistemic and aleatoric
uncertainty. Risk-aversion to epistemic uncertainty prevents distributional
shift, as areas not covered by the dataset have high epistemic uncertainty.
Risk-aversion to aleatoric uncertainty discourages actions that may result in
poor outcomes due to environment stochasticity. Our experiments show that our
algorithm achieves competitive performance on deterministic benchmarks, and
outperforms existing approaches for risk-sensitive objectives in stochastic
domains.
