  We establish geometric and topological properties of the space of value
functions in finite state-action Markov decision processes. Our main
contribution is the characterization of the nature of its shape: a general
polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several
properties of the structural relationship between policies and value functions
including the line theorem, which shows that the value functions of policies
constrained on all but one state describe a line segment. Finally, we use this
novel perspective to introduce visualizations to enhance the understanding of
the dynamics of reinforcement learning algorithms.
