<p>See my comment above, but it is possible to assume that B acquires the same reward value as the original reward, so B can then be treated as a reward, which in turn brings A closer to the reward event (now at time B). Does that make sense? In other words, the addition of B allows the reward value to accrue to B and thus brings A closer to reward.</p>

CC BY-SA 3.0
2018-03-31T01:02:05.237Z
2018-03-31T01:02:05.237Z
sometimes_sci
<p>I'm reading <a href="http://gershmanlab.webfactional.com/pubs/RL_causal.pdf" rel="noreferrer">Reinforcement learning and causal models</a> by Sam Gershman, which states that TD-learning provides an account of second-order conditioning that provides an explanation for the phenomenon occurred in trace learning (pg. 7, 3rd paragraph). The idea is as follows:</p>

<p>In trace conditioning, a reward is presented after some cue A with some relatively large time delay. If a second cue, B, is presented during this gap, this facilitates the learning of a conditional response to A. He says that this is because B receives a large positive value which generates a large positive prediction error at the offset of A. I do not see why this is true. If you look at TD-learning, the values of A and B should strictly be a function of their temporal distance from the reward, i.e. $$\gamma^{T-t} r_T$$, where $t$ is the current time and $T$ is the time of the reward. He goes on to say that this is related to how the prediction error is changed with the presence of B. Could somebody explain to me why having B increases the prediction error? </p>

CC BY-SA 3.0
2017-05-21T07:08:46.510Z
Hi, welcome at CogSci and very great question! Looking forward to an answer :)
Robin Kramer-ten Have
CC BY-SA 3.0
2018-03-31T01:01:12.540Z
I'm not sure there is a definitive answer to this, since in my experience the way the model explains phenomena such as second-order conditioning is somewhat up to the modeller. It's then an empirical question as to whether that explanation is correct.
sometimes_sci
<mathematical-psychology><reinforcement-learning>
How does TD-learning explain trace conditioning?
wcarvalho
# How does TD-learning explain trace conditioning?

(Asked by: username_0 on 2017-05-20)

I'm reading [Reinforcement learning and causal models](http://gershmanlab.webfactional.com/pubs/RL_causal.pdf) by Sam Gershman, which states that TD-learning provides an account of second-order conditioning that provides an explanation for the phenomenon occurred in trace learning (pg. 7, 3rd paragraph). The idea is as follows:

In trace conditioning, a reward is presented after some cue A with some relatively large time delay. If a second cue, B, is presented during this gap, this facilitates the learning of a conditional response to A. He says that this is because B receives a large positive value which generates a large positive prediction error at the offset of A. I do not see why this is true. If you look at TD-learning, the values of A and B should strictly be a function of their temporal distance from the reward, i.e. $$\gamma^{T-t} r\_T$$, where $t$ is the current time and $T$ is the time of the reward. He goes on to say that this is related to how the prediction error is changed with the presence of B. Could somebody explain to me why having B increases the prediction error?

**Tags:** mathematical-psychology, reinforcement-learning

> **Comments:**
> **username_1**: Hi, welcome at CogSci and very great question! Looking forward to an answer :)  
> **username_2**: I'm not sure there is a definitive answer to this, since in my experience the way the model explains phenomena such as second-order conditioning is somewhat up to the modeller. It's then an empirical question as to whether that explanation is correct.  

---

## 1 answers

### Answer by username_2 on 2018-03-31. Score: 1

See my comment above, but it is possible to assume that B acquires the same reward value as the original reward, so B can then be treated as a reward, which in turn brings A closer to the reward event (now at time B). Does that make sense? In other words, the addition of B allows the reward value to accrue to B and thus brings A closer to reward.


---

