  The space of value functions is a fundamental concept in reinforcement
learning. Characterizing its geometric properties may provide insights for
optimization and representation. Existing works mainly focus on the value space
for Markov Decision Processes (MDPs). In this paper, we study the geometry of
the robust value space for the more general Robust MDPs (RMDPs) setting, where
transition uncertainties are considered. Specifically, since we find it hard to
directly adapt prior approaches to RMDPs, we start with revisiting the
non-robust case, and introduce a new perspective that enables us to
characterize both the non-robust and robust value space in a similar fashion.
The key of this perspective is to decompose the value space, in a state-wise
manner, into unions of hypersurfaces. Through our analysis, we show that the
robust value space is determined by a set of conic hypersurfaces, each of which
contains the robust values of all policies that agree on one state.
Furthermore, we find that taking only extreme points in the uncertainty set is
sufficient to determine the robust value space. Finally, we discuss some other
aspects about the robust value space, including its non-convexity and policy
agreement on multiple states.
