<p>Your question is, in essence, a request for a mathematical model of <a href="http://scholarpedia.org/article/Temporal_difference_learning" rel="nofollow">Temporal Difference Reinforcement Learning</a>.  In a nutshell, temporal difference models add a notion of time to reinforcement learning models, which describe reinforcement learning as a comparison between what was expected and what actually occurred.</p>

<p>I think most of your answers can be found in the article above.  =)</p>

<p><strong>Edit</strong> re: my insistence on mathematical models providing the answer to your question:</p>

<p>Just to be clear, mathematical models don't exist in a vaccuum.  <em>Good</em> models are:</p>

<ol>
<li>built from constraints imposed by experimental observation</li>
<li>tractable; that is, they reflect the (presumed) underlying implementation</li>
</ol>

<p>Case and point, the <a href="http://www.scholarpedia.org/article/Rescorla-Wagner_model" rel="nofollow">Rescorla-Wagner model</a>'s<sup>1</sup> <code>delta</code> (prediction error) parameter has been <a href="http://jn.physiology.org/content/80/1/1" rel="nofollow">correlated to the phasic activity of dopaminergic neurons</a>.  This is precisely why the model is so hugely successful; it describes not only the experimental data, but the apparent neurological underpinnings, at least superficially.</p>

<p>Note that this is a very important kind of validity for neurological models, so much so, in fact, that there is an entire class of neuroimaging papers dedicated to finding functional activations that correspond to various parameters in such high-quality models.  <a href="http://www.sciencedirect.com/science/article/pii/S0896627303001697" rel="nofollow">Exhibit B.</a></p>

<p>I think your confusion arises from a lack of familiarity with the modelling approach to cognitive neuroscience (and that's okay! It's not obvious!), as well as a non-trivial distinction between the questions "what happens", "how does it happen" and "where does it happen".  You specifically asked <em>what</em> happens, which is the realm of formal mathematics.  <em>What happens</em>, as best we can tell, is that learning is driven by updating a prediction about what will happen concomitantly with (or shortly after) a stimulus, with what <em>actually happens when that stimulus is sensed</em>.  The models describe the exact nature of this interaction.  If you want to be more specific than this, you need to dive into the math.  If you want to know where or how these computations are performed, then that's another question entirely.</p>

<p><strong>1.</strong>  this model doesn't integrate any notion of time, contrary to the above TDRL model</p>

CC BY-SA 3.0
2014-06-30T12:21:23.060Z
But I'm looking for more than the matter of timing. I'd also like to know how our perception and state of mind influence the change. This seems mostly mathematical, with little mention of actual psychology.
user1999728
CC BY-SA 3.0
2014-06-30T17:17:04.237Z
@user1999728, In that case, either your question is unclear or you haven't understood the purpose of a mathematical model.  Your asking (1) how knowing the valence of a stimulus affects the organism's response, (2) what happens if you do two things concurrently and obtain a reward, (3) how the time course of stimulus-reward affects reinforcement.

These are all questions that are answered by mathematical models, as they formalize observed data patterns and enable us to make accurate predictions of behavior.  In other words, this *is* the answer to your question, but you have to read it =)
Louis Thibault
CC BY-SA 3.0
2014-06-30T17:41:21.540Z
All I see there is the math and algorithms behind reinforcement learning in machine learning, and a little mention of relevance to psychology and neuroscience. There is very little relevance to positive reinforcement in people. No empirical evidence whatsoever, and nothing about how our understanding of the task affects the outcome.
user1999728
CC BY-SA 3.0
2014-06-30T17:47:20.280Z
@user1999728, please see my edits.  These models are motivated by behavioral and neurophysiological data, the point being that they best describe the behavior of the organisms you're inquiring about.  Thus, if you want to know how various things affect various conditioning-driven behaviors, these "maths and algorithms" will provide the answer.  *That* is their relevance to psychology and neuroscience.
Louis Thibault
CC BY-SA 3.0
2014-06-30T17:58:00.397Z
well, I appreciate your attempts, and I understand the relevance, but I don't understand the model one bit. I don't understand how "change in a theoretical associative strength" is seen in the real world, I don't understand how to calculate $\alpha$ or $\beta$, how to measure maximum conditioning (or what that is), and so many more things.

I'm sorry, I assumed I'll only get results of empirical studies, which is what I want. I'll edit the post
user1999728
CC BY-SA 3.0
2014-06-30T21:22:33.583Z
@user1999728, I'm sorry I couldn't be more pedagogical =/  The thing is, this kind of stuff is going to pop up in any sort of empirical study because RL models *describe* behavior well, but don't offer a neurophysiological mechanism (and again, these studies belong to the realm of the *how* and not the *what*).  I'd suggest you begin by familiarizing yourself with the Rescorla-Wagner model.  Start from the top, and come back with specific questions (e.g.: "what is the α value and how do I calculate it?").  From there, TDRL should be much more approachable.
Louis Thibault
CC BY-SA 3.0
2014-06-30T11:31:57.203Z
2014-06-30T21:23:17.270Z
2014-06-30T21:23:17.270Z
Louis Thibault
<p>I've been thinking about positive reinforcement occasionally, and I've always found myself wondering about various things that can influence it.</p>

<p>So, I'm looking for information (results of empirical studies) about the variables affecting positive reinforcement, in hopes of finding answers to my questions and questions that I haven't even thought of, such as:</p>

<ol>
<li>How does knowing what the positive stimulus is for affect the change? What happens if I did two things, and I don't know which one of them led to the positive stimulus? e.g. I solved a math problem and won some game, and then I received a prize, and I don't know for which one of these things</li>
</ol>

<p>What happens If (as far as I'm concerned) I did nothing, and then I receive a prize?</p>

<ol>
<li><p>If I receive a prize for something I did an hour earlier, for example, will it still change my behavior?</p></li>
<li><p>Does receiving a prize while I do a task has a stronger effect? weaker?</p></li>
</ol>

<p>Any other interesting questions (as well as answers to them) are welcome </p>

<p>(And since someone is probably going to make a few changes and add tags - thank you and forgive me. English is not my native language, and I don't have much of a background in psychology)</p>

<cognitive-psychology><positive-psychology>
Variables in positive reinforcement in human beings
user1999728
# Variables in positive reinforcement in human beings

(Asked by: username_0 on 2014-06-30)

I've been thinking about positive reinforcement occasionally, and I've always found myself wondering about various things that can influence it.

So, I'm looking for information (results of empirical studies) about the variables affecting positive reinforcement, in hopes of finding answers to my questions and questions that I haven't even thought of, such as:

1. How does knowing what the positive stimulus is for affect the change? What happens if I did two things, and I don't know which one of them led to the positive stimulus? e.g. I solved a math problem and won some game, and then I received a prize, and I don't know for which one of these things

What happens If (as far as I'm concerned) I did nothing, and then I receive a prize?

1. If I receive a prize for something I did an hour earlier, for example, will it still change my behavior?
2. Does receiving a prize while I do a task has a stronger effect? weaker?

Any other interesting questions (as well as answers to them) are welcome

(And since someone is probably going to make a few changes and add tags - thank you and forgive me. English is not my native language, and I don't have much of a background in psychology)

**Tags:** cognitive-psychology, positive-psychology


---

## 1 answers

### Answer by username_1 on 2014-06-30. Score: 2

Your question is, in essence, a request for a mathematical model of [Temporal Difference Reinforcement Learning](http://scholarpedia.org/article/Temporal_difference_learning). In a nutshell, temporal difference models add a notion of time to reinforcement learning models, which describe reinforcement learning as a comparison between what was expected and what actually occurred.

I think most of your answers can be found in the article above. =)

**Edit** re: my insistence on mathematical models providing the answer to your question:

Just to be clear, mathematical models don't exist in a vaccuum. *Good* models are:

1. built from constraints imposed by experimental observation
2. tractable; that is, they reflect the (presumed) underlying implementation

Case and point, the [Rescorla-Wagner model](http://www.scholarpedia.org/article/Rescorla-Wagner_model)'s1 `delta` (prediction error) parameter has been [correlated to the phasic activity of dopaminergic neurons](http://jn.physiology.org/content/80/1/1). This is precisely why the model is so hugely successful; it describes not only the experimental data, but the apparent neurological underpinnings, at least superficially.

Note that this is a very important kind of validity for neurological models, so much so, in fact, that there is an entire class of neuroimaging papers dedicated to finding functional activations that correspond to various parameters in such high-quality models. [Exhibit B.](http://www.sciencedirect.com/science/article/pii/S0896627303001697)

I think your confusion arises from a lack of familiarity with the modelling approach to cognitive neuroscience (and that's okay! It's not obvious!), as well as a non-trivial distinction between the questions "what happens", "how does it happen" and "where does it happen". You specifically asked *what* happens, which is the realm of formal mathematics. *What happens*, as best we can tell, is that learning is driven by updating a prediction about what will happen concomitantly with (or shortly after) a stimulus, with what *actually happens when that stimulus is sensed*. The models describe the exact nature of this interaction. If you want to be more specific than this, you need to dive into the math. If you want to know where or how these computations are performed, then that's another question entirely.

**1.** this model doesn't integrate any notion of time, contrary to the above TDRL model

> **Comments:**
> **username_0**: But I'm looking for more than the matter of timing. I'd also like to know how our perception and state of mind influence the change. This seems mostly mathematical, with little mention of actual psychology.  
> **username_1**: @username_0, In that case, either your question is unclear or you haven't understood the purpose of a mathematical model.  Your asking (1) how knowing the valence of a stimulus affects the organism's response, (2) what happens if you do two things concurrently and obtain a reward, (3) how the time course of stimulus-reward affects reinforcement.  These are all questions that are answered by mathematical models, as they formalize observed data patterns and enable us to make accurate predictions of behavior.  In other words, this *is* the answer to your question, but you have to read it =)  
> **username_0**: All I see there is the math and algorithms behind reinforcement learning in machine learning, and a little mention of relevance to psychology and neuroscience. There is very little relevance to positive reinforcement in people. No empirical evidence whatsoever, and nothing about how our understanding of the task affects the outcome.  
> **username_1**: @username_0, please see my edits.  These models are motivated by behavioral and neurophysiological data, the point being that they best describe the behavior of the organisms you're inquiring about.  Thus, if you want to know how various things affect various conditioning-driven behaviors, these "maths and algorithms" will provide the answer.  *That* is their relevance to psychology and neuroscience.  
> **username_0**: well, I appreciate your attempts, and I understand the relevance, but I don't understand the model one bit. I don't understand how "change in a theoretical associative strength" is seen in the real world, I don't understand how to calculate $\alpha$ or $\beta$, how to measure maximum conditioning (or what that is), and so many more things.  I'm sorry, I assumed I'll only get results of empirical studies, which is what I want. I'll edit the post  
> **username_1**: @username_0, I'm sorry I couldn't be more pedagogical =/  The thing is, this kind of stuff is going to pop up in any sort of empirical study because RL models *describe* behavior well, but don't offer a neurophysiological mechanism (and again, these studies belong to the realm of the *how* and not the *what*).  I'd suggest you begin by familiarizing yourself with the Rescorla-Wagner model.  Start from the top, and come back with specific questions (e.g.: "what is the α value and how do I calculate it?").  From there, TDRL should be much more approachable.  

---

