  We address the challenge of quantifying Bayesian uncertainty and
incorporating it in offline use cases of finite-state Markov Decision Processes
(MDPs) with unknown dynamics. Our approach provides a principled method to
disentangle epistemic and aleatoric uncertainty, and a novel technique to find
policies that optimise Bayesian posterior expected value without relying on
strong assumptions about the MDP's posterior distribution. First, we utilise
standard Bayesian reinforcement learning methods to capture the posterior
uncertainty in MDP parameters based on available data. We then analytically
compute the first two moments of the return distribution across posterior
samples and apply the law of total variance to disentangle aleatoric and
epistemic uncertainties. To find policies that maximise posterior expected
value, we leverage the closed-form expression for value as a function of
policy. This allows us to propose a stochastic gradient-based approach for
solving the problem. We illustrate the uncertainty quantification and Bayesian
posterior value optimisation performance of our agent in simple, interpretable
gridworlds and validate it through ground-truth evaluations on synthetic MDPs.
Finally, we highlight the real-world impact and computational scalability of
our method by applying it to the AI Clinician problem, which recommends
treatment for patients in intensive care units and has emerged as a key use
case of finite-state MDPs with offline data. We discuss the challenges that
arise with Bayesian modelling of larger scale MDPs while demonstrating the
potential to apply our methods rooted in Bayesian decision theory into the real
world. We make our code available at
https://github.com/filippovaldettaro/finite-state-mdps .
