  We consider infinite-horizon Markov Decision Processes where parameters, such
as transition probabilities, are unknown and estimated from data. The popular
distributionally robust approach to addressing the parameter uncertainty can
sometimes be overly conservative. In this paper, we utilize the recently
proposed formulation, Bayesian risk Markov Decision Process (BR-MDP), to
address parameter (or epistemic) uncertainty in MDPs (Lin et al., 2022). To
solve the infinite-horizon BR-MDP with a class of convex risk measures, we
propose a computationally efficient approach of approximate bilevel difference
convex programming (ABDCP). The optimization is performed offline and produces
the optimal policy that is represented as a finite state controller with
desirable performance guarantees. We also demonstrate the empirical performance
of the infinite-horizon BR-MDP formulation and proposed algorithms.
