  In the world of stochastic control, especially in economics and engineering,
Markov Decision Processes (MDPs) can effectively model various stochastic
decision processes, from asset management to transportation optimization. These
underlying MDPs, upon closer examination, often reveal a specifically
constrained causal structure concerning the transition and reward dynamics. By
exploiting this structure, we can obtain a reduction in the causal
representation of the problem setting, allowing us to solve of the optimal
value function more efficiently. This work defines an MDP framework, the
\texttt{SD-MDP}, where we disentangle the causal structure of MDPs' transition
and reward dynamics, providing distinct partitions on the temporal causal
graph. With this stochastic reduction, the \texttt{SD-MDP} reflects a general
class of resource allocation problems. This disentanglement further enables us
to derive theoretical guarantees on the estimation error of the value function
under an optimal policy by allowing independent value estimation from Monte
Carlo sampling. Subsequently, by integrating this estimator into well-known
Monte Carlo planning algorithms, such as Monte Carlo Tree Search (MCTS), we
derive bounds on the simple regret of the algorithm. Finally, we quantify the
policy improvement of MCTS under the \texttt{SD-MDP} framework by demonstrating
that the MCTS planning algorithm achieves higher expected reward (lower costs)
under a constant simulation budget, on a tangible economic example based on
maritime refuelling.
