  Optimal policies in Markov decision processes (MDPs) are very sensitive to
model misspecification. This raises serious concerns about deploying them in
high-stake domains. Robust MDPs (RMDP) provide a promising framework to
mitigate vulnerabilities by computing policies with worst-case guarantees in
reinforcement learning. The solution quality of an RMDP depends on the
ambiguity set, which is a quantification of model uncertainties. In this paper,
we propose a new approach for optimizing the shape of the ambiguity sets for
RMDPs. Our method departs from the conventional idea of constructing a
norm-bounded uniform and symmetric ambiguity set. We instead argue that the
structure of a near-optimal ambiguity set is problem specific. Our proposed
method computes a weight parameter from the value functions, and these weights
then drive the shape of the ambiguity sets. Our theoretical analysis
demonstrates the rationale of the proposed idea. We apply our method to several
different problem domains, and the empirical results further furnish the
practical promise of weighted near-optimal ambiguity sets.
