  Markov Decision Processes (MDPs) are mathematical models of sequential
decision-making under uncertainty that have found applications in healthcare,
manufacturing, logistics, and others. In these models, a decision-maker
observes the state of a stochastic process and determines which action to take
with the goal of maximizing the expected total discounted rewards received. In
many applications, the state space of the true system is large and there may be
limited observations out of certain states to estimate the transition
probability matrix. To overcome this, modelers will aggregate the true states
into ``superstates" resulting in a smaller state space. This aggregation
process improves computational tractability and increases the number of
observations among superstates. Thus, the modeler's choice of state space leads
to a trade-off in transition probability estimates. While coarser
discretization of the state space gives more observations in each state to
estimate the transition probability matrix, this comes at the cost of precision
in the state characterization and resulting policy recommendations. In this
paper, we consider the implications of this modeling decision on the resulting
policies from MDPs for which the true model is expected to have a threshold
policy that is optimal. We analyze these MDPs and provide conditions under
which the aggregated MDP will also have an optimal threshold policy. Using a
simulation study, we explore the trade-offs between more fine and more coarse
aggregation. We explore the the show that there is the highest potential for
policy improvement on larger state spaces, but that aggregated MDPs are
preferable under limited data. We discuss how these findings the implications
of our findings for modelers who must select which state space design to use.
