<h2>Online/Offline learning</h2>

<p>Whether your algorithm learns online or offline is governed by when the agent's experience is able to contribute to its learning.</p>

<p>For example, consider value iteration algorithms for action values. If you use <strong>Monte Carlo</strong> methods to calculate your value updates, then you'd be performing <strong>offline</strong> learning as the agent will have to reach the end of an episode to calculate the full MC return for any action value. The full Monte Carlo return and the action value definition (expected return) is given by</p>

<p><span class="math-container">$G_t = R_{t+1} + γ R_{t+2} + γ^2 R_{t+3} + … + γ^{T-t-1} R_T$</span>, and</p>

<p><span class="math-container">$ q(s,a) = E[R_{t+1} + γ R_{t+2} + γ^2 R_{t+3} + … + γ^{T-t-1} R_T \ |\ S_t = s, A_t = a]$</span></p>

<p>where <span class="math-container">$T$</span> signifies last time step of the episode, and <span class="math-container">$\gamma$</span> is the discount factor</p>

<p>On the other hand if you use <strong>Temporal Difference</strong> methods, the action values can be updated right away by bootstrapping to already calculated values, making the learning <strong>online</strong>. We use the recursive definition of return to get recursive definition of value (expected return) in TD methods. </p>

<p><span class="math-container">$G_t = R_{t+1} + γ G_{t+1}$</span>, and</p>

<p><span class="math-container">$ q(s,a) = E[R_{t+1} + γ\ q(S_{t+1}, a') \ |\ S_t = s, A_t = a]$</span></p>

<h2>On-policy/Off-policy learning</h2>

<p>This depends on the update function you're using to move your state/action value estimates towards the real expected return of the defintion.</p>

<p>For example, in case of <strong>SARSA</strong> (update action values to evaluate behavior policy) with 1-step TD updates, we have the update function as</p>

<p><span class="math-container">$Q_{t+1}(S_t, A_t) = Q_t(S_t, A_t) + α (R_{t+1} + γ\ Q_t(S_{t+1}, A_{t+1}) - Q_t(S_t, A_t))$</span></p>

<p>Since this method uses the action value for that action which is actually performed in the next state (<span class="math-container">$a' = A_{t+1}$</span> for <span class="math-container">$q(s,a)$</span> estimate), the estimates will converge to action values for the behavior policy, and not for the optimal policy. Hence, it is a form of <strong>on-policy</strong> learning.</p>

<p>Now if you consider <strong>Q-learning</strong>, the update function is</p>

<p><span class="math-container">$Q_{t+1}(S_t, A_t) = Q_t(S_t, A_t) + α (R_{t+1} + γ\ max_{a'} Q_t(S_{t+1}, a') - Q_t(S_t, A_t))$</span></p>

<p>Here, we don’t need to know what action is to be taken from the next state, <span class="math-container">$S_{t+1}$</span>.
We use the maximum q-value across all actions for the next state to calculate our estimate (<span class="math-container">$a' = argmax_{a'} Q_t(S_{t+1}, a')$</span> for <span class="math-container">$q(s,a)$</span> estimate).
Because of this, the agent learns q-values for the optimal or greedy policy, regardless of the one it is following currently (<span class="math-container">$A_{t+1}$</span> doesn’t affect the q-value update). Hence, it is an example of <strong>off-policy</strong> learning.</p>

<p>Given this information, an example of <em>On-policy/offline evaluation</em> would be simple <strong>Monte Carlo learning</strong>, where action values are calculated as the average of Monte Carlo returns over multiple episodes.</p>

<p>And an example of <em>Off-policy/online evaluation</em> would be <strong>Q-learning</strong> with 1-step TD updates. Because of bootstrapping in TD updates, it is online, and due to the nature of the update function (given above), it estimates optimal values.</p>

CC BY-SA 4.0
2019-02-18T11:29:23.897Z
2019-02-18T11:29:23.897Z
Dwait Bhatt
<p>In the context of the following question:
<a href="https://cstheory.stackexchange.com/questions/3431/off-policy-and-offline-policy-reinforcement-learning">off-policy and offline policy reinforcement learning</a>
, it can be concluded that off-policy/on-policy learning can be orthogonal to an online/offline sampling scenario.</p>

<p>I am having trouble connecting these concepts to the idea of evaluating an RL approach (target/behavior policy) aimed to be deployed in a real-world environment (e.g. a web application).</p>

<p>In this case, I believe that the online evaluation would be to test the RL approach when deployed and that the offline evaluation would be to use historical data or a simulation of the real-world environment.</p>

<p>Following this assumption, examples of the different evaluation combinations can be:</p>

<ul>
<li><em>Off-policy/offline evaluation:</em> evaluate the target policy using a historical dataset collected with a policy other than the target policy.</li>
<li><em>On-policy/online evaluation:</em> experience used to evaluate is collected by following the target policy sampling directly from the real-world environment.</li>
</ul>

<p>But it is hard for me to think of examples for the other combinations:</p>

<ul>
<li><em>On-policy/offline evaluation:</em> data used to evaluate the target policy is collected by following the target policy on a simulation of the real-world environment. Does this make sense?</li>
<li><em>Off-policy/online evaluation:</em> I can't think of an example that makes sense, i.e., if the target policy can be evaluated directly with the real-world environment, why use a different behavior/target policy to sample experience? </li>
</ul>

<p>Note that my question is in the context of evaluation, i.e. the goal is only to evaluate the target policy and not to learn a new policy.</p>

<p>I am wondering if any of these ideas make sense and if anyone can think of an example for the case of Off-policy/online evaluation?</p>

<lg.learning><reinforcement-learning>
On-policy/Off-policy Offline/Online Evaluation: Which would be an example of Online Off-Policy Evaluation?
Andrea Barraza
# On-policy/Off-policy Offline/Online Evaluation: Which would be an example of Online Off-Policy Evaluation?

(Asked by: username_0 on 2018-09-24)

In the context of the following question:
[off-policy and offline policy reinforcement learning](https://cstheory.stackexchange.com/questions/3431/off-policy-and-offline-policy-reinforcement-learning)
, it can be concluded that off-policy/on-policy learning can be orthogonal to an online/offline sampling scenario.

I am having trouble connecting these concepts to the idea of evaluating an RL approach (target/behavior policy) aimed to be deployed in a real-world environment (e.g. a web application).

In this case, I believe that the online evaluation would be to test the RL approach when deployed and that the offline evaluation would be to use historical data or a simulation of the real-world environment.

Following this assumption, examples of the different evaluation combinations can be:

* *Off-policy/offline evaluation:* evaluate the target policy using a historical dataset collected with a policy other than the target policy.
* *On-policy/online evaluation:* experience used to evaluate is collected by following the target policy sampling directly from the real-world environment.

But it is hard for me to think of examples for the other combinations:

* *On-policy/offline evaluation:* data used to evaluate the target policy is collected by following the target policy on a simulation of the real-world environment. Does this make sense?
* *Off-policy/online evaluation:* I can't think of an example that makes sense, i.e., if the target policy can be evaluated directly with the real-world environment, why use a different behavior/target policy to sample experience?

Note that my question is in the context of evaluation, i.e. the goal is only to evaluate the target policy and not to learn a new policy.

I am wondering if any of these ideas make sense and if anyone can think of an example for the case of Off-policy/online evaluation?

**Tags:** lg.learning, reinforcement-learning

---

## 1 answers

### Answer by username_1 on 2019-02-18. Score: 1

## Online/Offline learning

Whether your algorithm learns online or offline is governed by when the agent's experience is able to contribute to its learning.

For example, consider value iteration algorithms for action values. If you use **Monte Carlo** methods to calculate your value updates, then you'd be performing **offline** learning as the agent will have to reach the end of an episode to calculate the full MC return for any action value. The full Monte Carlo return and the action value definition (expected return) is given by

$G\_t = R\_{t+1} + γ R\_{t+2} + γ^2 R\_{t+3} + … + γ^{T-t-1} R\_T$, and

$ q(s,a) = E[R\_{t+1} + γ R\_{t+2} + γ^2 R\_{t+3} + … + γ^{T-t-1} R\_T \ |\ S\_t = s, A\_t = a]$

where $T$ signifies last time step of the episode, and $\gamma$ is the discount factor

On the other hand if you use **Temporal Difference** methods, the action values can be updated right away by bootstrapping to already calculated values, making the learning **online**. We use the recursive definition of return to get recursive definition of value (expected return) in TD methods.

$G\_t = R\_{t+1} + γ G\_{t+1}$, and

$ q(s,a) = E[R\_{t+1} + γ\ q(S\_{t+1}, a') \ |\ S\_t = s, A\_t = a]$

## On-policy/Off-policy learning

This depends on the update function you're using to move your state/action value estimates towards the real expected return of the defintion.

For example, in case of **SARSA** (update action values to evaluate behavior policy) with 1-step TD updates, we have the update function as

$Q\_{t+1}(S\_t, A\_t) = Q\_t(S\_t, A\_t) + α (R\_{t+1} + γ\ Q\_t(S\_{t+1}, A\_{t+1}) - Q\_t(S\_t, A\_t))$

Since this method uses the action value for that action which is actually performed in the next state ($a' = A\_{t+1}$ for $q(s,a)$ estimate), the estimates will converge to action values for the behavior policy, and not for the optimal policy. Hence, it is a form of **on-policy** learning.

Now if you consider **Q-learning**, the update function is

$Q\_{t+1}(S\_t, A\_t) = Q\_t(S\_t, A\_t) + α (R\_{t+1} + γ\ max\_{a'} Q\_t(S\_{t+1}, a') - Q\_t(S\_t, A\_t))$

Here, we don’t need to know what action is to be taken from the next state, $S\_{t+1}$.
We use the maximum q-value across all actions for the next state to calculate our estimate ($a' = argmax\_{a'} Q\_t(S\_{t+1}, a')$ for $q(s,a)$ estimate).
Because of this, the agent learns q-values for the optimal or greedy policy, regardless of the one it is following currently ($A\_{t+1}$ doesn’t affect the q-value update). Hence, it is an example of **off-policy** learning.

Given this information, an example of *On-policy/offline evaluation* would be simple **Monte Carlo learning**, where action values are calculated as the average of Monte Carlo returns over multiple episodes.

And an example of *Off-policy/online evaluation* would be **Q-learning** with 1-step TD updates. Because of bootstrapping in TD updates, it is online, and due to the nature of the update function (given above), it estimates optimal values.


---

