<p>They are the same in the two equations.  See equation 9.3:  v = w*u.  w and u are bolded in that equation to indicate that they are vectors.  In equation 9.3, they are vectors of simultaneously presented stimuli, but you can apply the same equation if u is a vector of a single stimulus over time and w are the weights of each time step of that stimulus.  Indeed, a linear filter is simply a way to weight different time steps of a time-varying input.  Note that v in equation 9.3 is a single value, whereas v in equation 9.6 is a function of t, i.e. it has a different value at each time point depending on what part of u is currently occurring. </p>

CC BY-SA 3.0
2018-02-23T19:39:59.143Z
Thanks honi, I get it now!  -- At any time t, when calculating the prediction v(t), w(z) is used to weigh the contribution of u(t-z), the value that the stimulus had z ticks earlier.
Nasorenga
CC BY-SA 3.0
2018-02-23T21:05:37.813Z
yep.  do you mind accepting my answer if it resolved your problem?
honi
CC BY-SA 3.0
2018-02-23T01:15:04.363Z
2018-02-23T01:15:04.363Z
honi
<p>In the Temporal difference learning algorithm (TD-learning), an agent seeks to predict the total value of future rewards that will be received during the current trial. The agent updates the prediction as the trial progresses, based on conditioned stimuli (CS) that have been seen so far. Specifically, the prediction is calculated as a weighted sum of the intensities of past CS presentation. At time $t$, the prediction is calculated as </p>

<p>$$v(t) = \sum_{z=0}^t w(z)u(t-z)$$</p>

<p>where $v(t)$ is the prediction at time $t$, $u(t)$ is the CS intensity at time $t$, and $w$ is an array of weights.</p>

<p>My question is: why are we summing the terms $w(z)u(t-z)$, rather than $w(z)u(z)$, i.e. why isn't each weight associated with the CS strength at a particular time?</p>

<p>[Thanks to user honi's reply, I subsequently understood that each element of the weight array is associated with an elapsed time since seeing the CS. For example, $w(3)$ is the weight given to the CS strength experienced three units of time ago.]</p>

<hr>

<p>[Original phrasing of the question: I am reading Theoretical Neuroscience by Dayan and Abbot, and I am confused by the use of $w(t)$ in equations (9.6) and (9.7). In the former, it is a window function (linear filter) and in the latter it is the weight function. Is this (a) an unfortunate use of the same name for two different things, or (b) is the weight function really used as a linear filter to calculate $v(t)$?]</p>

CC BY-SA 3.0
2018-02-21T23:03:54.400Z
Welcome. Do realize not everyone owns the referenced book. Mentioning formula numbers is not really helpful. Instead, a clear explanation of the context, acronyms and symbols is necessary for folks to understand your question. Personally, I'm at a loss at what you are asking.
AliceD
CC BY-SA 3.0
2018-02-23T19:44:18.197Z
Thanks for the advice, AliceD. - I wasn't under the illusion that everyone owned this book, just hoping to connect with one person who did and could help me out - and it worked! :-)
Nasorenga
CC BY-SA 3.0
2018-02-23T19:55:36.057Z
Do realize any post should be I interesting for the community at large. The fact you are helped out doesn't mean it's on topic here. @Honi perhaps you can update the question?
AliceD
<neuroscience><theoretical-neuroscience><computational-modeling>
In TD learning, is the weight function used as a linear filter?
Nasorenga
# In TD learning, is the weight function used as a linear filter?

(Asked by: username_0 on 2018-02-21)

In the Temporal difference learning algorithm (TD-learning), an agent seeks to predict the total value of future rewards that will be received during the current trial. The agent updates the prediction as the trial progresses, based on conditioned stimuli (CS) that have been seen so far. Specifically, the prediction is calculated as a weighted sum of the intensities of past CS presentation. At time $t$, the prediction is calculated as

$$v(t) = \sum\_{z=0}^t w(z)u(t-z)$$

where $v(t)$ is the prediction at time $t$, $u(t)$ is the CS intensity at time $t$, and $w$ is an array of weights.

My question is: why are we summing the terms $w(z)u(t-z)$, rather than $w(z)u(z)$, i.e. why isn't each weight associated with the CS strength at a particular time?

[Thanks to user username_2's reply, I subsequently understood that each element of the weight array is associated with an elapsed time since seeing the CS. For example, $w(3)$ is the weight given to the CS strength experienced three units of time ago.]

---

[Original phrasing of the question: I am reading Theoretical Neuroscience by Dayan and Abbot, and I am confused by the use of $w(t)$ in equations (9.6) and (9.7). In the former, it is a window function (linear filter) and in the latter it is the weight function. Is this (a) an unfortunate use of the same name for two different things, or (b) is the weight function really used as a linear filter to calculate $v(t)$?]

**Tags:** neuroscience, theoretical-neuroscience, computational-modeling

> **Comments:**
> **username_1**: Welcome. Do realize not everyone owns the referenced book. Mentioning formula numbers is not really helpful. Instead, a clear explanation of the context, acronyms and symbols is necessary for folks to understand your question. Personally, I'm at a loss at what you are asking.  
> **username_0**: Thanks for the advice, username_1. - I wasn't under the illusion that everyone owned this book, just hoping to connect with one person who did and could help me out - and it worked! :-)  
> **username_1**: Do realize any post should be I interesting for the community at large. The fact you are helped out doesn't mean it's on topic here. @Honi perhaps you can update the question?  

---

## 1 answers

### Answer by username_2 on 2018-02-23. Score: 2 | Accepted answer

They are the same in the two equations. See equation 9.3: v = w\*u. w and u are bolded in that equation to indicate that they are vectors. In equation 9.3, they are vectors of simultaneously presented stimuli, but you can apply the same equation if u is a vector of a single stimulus over time and w are the weights of each time step of that stimulus. Indeed, a linear filter is simply a way to weight different time steps of a time-varying input. Note that v in equation 9.3 is a single value, whereas v in equation 9.6 is a function of t, i.e. it has a different value at each time point depending on what part of u is currently occurring.

> **Comments:**
> **username_0**: Thanks username_2, I get it now!  -- At any time t, when calculating the prediction v(t), w(z) is used to weigh the contribution of u(t-z), the value that the stimulus had z ticks earlier.  
> **username_2**: yep.  do you mind accepting my answer if it resolved your problem?  

---

