<p>The set of optimal policies for one MDP may be completely different than for another MDP even if the two MPDs differ only in the rewards and/or transition probabilities.</p>

<p>One thing you might try in order to save resources in some cases when you change the MDP (e.g. when the set of optimal policies does not change too much) is to use policy iteration to find optimal policies and to start it for the new MDP from an optimal policy for the old MDP. Theoretically, you should bear in mind that there are exponential time examples for policy iteration that start and finish at policies that differ in only one action, but these are pathological examples that will almost certainly not arise in practice.</p>

<p>You may find some relevant information if you search for "sensitivity analysis" or "perturbation analysis" for MDPs.</p>

CC BY-SA 3.0
2014-05-08T21:47:56.353Z
2016-02-16T19:58:00.880Z
2016-02-16T19:58:00.880Z
Rahul Savani
<p>I was using MDP on my work to make optimal decision. I used discrete time, finite state MDP. I assumed that I will have an initial parameters, like the Reward/Cost, state transition probabilities and actions, to make the optimal decisions. Then, I used the value iteration algorithm to find the expected values for each possible state under the given MDP configuration.</p>

<p>Here is my problem. As I understand, after getting the optimal policy from Value Iteration Algorithm, the policy will stay valid under the initial MDP configuration. However, I believe the policy must be adjusted whenever there is change on MDP configurations (specifically change in Reward/Cost and Transition probabilities). So, I am looking for the best solution that adjust the policy instead of recalculating the policy again using VIA under the new MDP configuration.</p>

<p>My work assumes the states and action doesn't change, only the cost and transition probability changes. Please suggest me any solution or reference to similar works. Thanks.</p>

<pr.probability><markov-chains>
Policy Adjustment in Markov Decision Process
Mati Bekuma
# Policy Adjustment in Markov Decision Process

(Asked by: username_0 on 2014-05-08)

I was using MDP on my work to make optimal decision. I used discrete time, finite state MDP. I assumed that I will have an initial parameters, like the Reward/Cost, state transition probabilities and actions, to make the optimal decisions. Then, I used the value iteration algorithm to find the expected values for each possible state under the given MDP configuration.

Here is my problem. As I understand, after getting the optimal policy from Value Iteration Algorithm, the policy will stay valid under the initial MDP configuration. However, I believe the policy must be adjusted whenever there is change on MDP configurations (specifically change in Reward/Cost and Transition probabilities). So, I am looking for the best solution that adjust the policy instead of recalculating the policy again using VIA under the new MDP configuration.

My work assumes the states and action doesn't change, only the cost and transition probability changes. Please suggest me any solution or reference to similar works. Thanks.

**Tags:** pr.probability, markov-chains

---

## 1 answers

### Answer by username_1 on 2014-05-08. Score: 4 | Accepted answer

The set of optimal policies for one MDP may be completely different than for another MDP even if the two MPDs differ only in the rewards and/or transition probabilities.

One thing you might try in order to save resources in some cases when you change the MDP (e.g. when the set of optimal policies does not change too much) is to use policy iteration to find optimal policies and to start it for the new MDP from an optimal policy for the old MDP. Theoretically, you should bear in mind that there are exponential time examples for policy iteration that start and finish at policies that differ in only one action, but these are pathological examples that will almost certainly not arise in practice.

You may find some relevant information if you search for "sensitivity analysis" or "perturbation analysis" for MDPs.


---

