<p>I assume that you know what "policy evaluation" means. Briefly speaking, it refers to the task of estimating the value of a given policy.</p>

<p>In the RL literature, the off-policy scenario refers to the situation that the policy you want to evaluate is different from the data generating policy. In other words, you behave in one way, but you want to evaluate the value of another policy. This should be contrasted with the on-policy scenario that the policy you want to evaluate is the same policy that generates (or generated) data.</p>

<p>The offline sampling scenario (and not "offline policy") is the scenario that you already have some samples and now you want to perform tasks like policy evaluation. In this scenario, the agent cannot have any further interaction with the environment. In other words, the data are given and fixed. This should be contrasted with the online sampling scenario that the agent actually interacts with the environment.</p>

<p>These two are orthogonal. So it is possible to have a on-policy/off-policy offline/online scenario.</p>

<p>Hope my short discussion has helped. You may also take a look at Sutton and Barto's textbook.</p>

<p>Amir massoud</p>

CC BY-SA 2.5
2010-12-04T01:13:25.457Z
2010-12-04T01:13:25.457Z
Amir-massoud Farahmand
<p>What's the difference between</p>

<blockquote>
  <p>off-policy reinforcement learning
  algorithms</p>
</blockquote>

<p>and</p>

<blockquote>
  <p>offline policy reinforcement learning
  algorithms ?</p>
</blockquote>

<p>Or do they mean the same thing ?
thanks</p>

CC BY-SA 2.5
2010-12-01T14:38:48.173Z
Please make your question self-contained and provide background and motivation for your question. See [How to ask a good question](http://meta.cstheory.stackexchange.com/questions/300/how-to-ask-a-good-question).
Kaveh
CC BY-SA 2.5
2010-12-01T16:17:44.710Z
to be more concrete: provide a short summary or link to description of RL, and also link to or explain/summarize the context of the two different kinds of RL you're asking about.
Suresh Venkat
<lg.learning><machine-learning>
off-policy and offline policy reinforcement learning
aneuryzm
# off-policy and offline policy reinforcement learning

(Asked by: username_0 on 2010-12-01)

What's the difference between

> off-policy reinforcement learning
> algorithms

and

> offline policy reinforcement learning
> algorithms ?

Or do they mean the same thing ?
thanks

**Tags:** lg.learning, machine-learning
> **Comments:**
> **username_1**: Please make your question self-contained and provide background and motivation for your question. See [How to ask a good question](http://meta.cstheory.stackexchange.com/questions/300/how-to-ask-a-good-question).  
> **username_2**: to be more concrete: provide a short summary or link to description of RL, and also link to or explain/summarize the context of the two different kinds of RL you're asking about.  

---

## 1 answers

### Answer by username_3 on 2010-12-04. Score: 10 | Accepted answer

I assume that you know what "policy evaluation" means. Briefly speaking, it refers to the task of estimating the value of a given policy.

In the RL literature, the off-policy scenario refers to the situation that the policy you want to evaluate is different from the data generating policy. In other words, you behave in one way, but you want to evaluate the value of another policy. This should be contrasted with the on-policy scenario that the policy you want to evaluate is the same policy that generates (or generated) data.

The offline sampling scenario (and not "offline policy") is the scenario that you already have some samples and now you want to perform tasks like policy evaluation. In this scenario, the agent cannot have any further interaction with the environment. In other words, the data are given and fixed. This should be contrasted with the online sampling scenario that the agent actually interacts with the environment.

These two are orthogonal. So it is possible to have a on-policy/off-policy offline/online scenario.

Hope my short discussion has helped. You may also take a look at Sutton and Barto's textbook.

Amir massoud


---

