  Robust MDPs (RMDPs) can be used to compute policies with provable worst-case
guarantees in reinforcement learning. The quality and robustness of an RMDP
solution are determined by the ambiguity set---the set of plausible transition
probabilities---which is usually constructed as a multi-dimensional confidence
region. Existing methods construct ambiguity sets as confidence regions using
concentration inequalities which leads to overly conservative solutions. This
paper proposes a new paradigm that can achieve better solutions with the same
robustness guarantees without using confidence regions as ambiguity sets. To
incorporate prior knowledge, our algorithms optimize the size and position of
ambiguity sets using Bayesian inference. Our theoretical analysis shows the
safety of the proposed method, and the empirical results demonstrate its
practical promise.
