<p>I'm not sure I fully understand your design; perhaps you can clarify what you want your network to learn, why TD-learning "isn't cutting it", and what you mean by 'reinforcement' and  'prediction' learning. In particular, TD-learning <em>is</em> a reinforcement learning model, and it <em>does</em> reward based on predicted (and not just observed) outcomes. However, you seem to be describing reinforcement and prediction learning as orthogonal models, so again, I'm not sure I understand correctly.</p>

<p>As a general suggestion, you might consider using an Elman/Jordan network (e.g. recurrent neural network/RNN). Rather than relying on knowledge of only the current state to make a prediction about the next state, RNN can learn to recognize sequences of events. This is especially useful for predicting future states in a task that unfolds over time (e.g. [1]). I suggest this mostly because you say your task is a 'real-time control' task, but without more knowledge of your task I don't really know if this is appropriate.</p>

<p>As for your suggestion of using two different learning mechanisms to modify a single set of weights-- I don't have the answer, but it seems counterintuitive to me. You're using two different optimization techniques on a single set of data. If the techniques disagree, your network will probably never learn its connection weights. If they agree (i.e., converge on the same answer) then I'm not certain you're adding any value by having two learning mechanisms.</p>

<p>[1] Elman (1990). Finding structure in time. Cognitive Science, 14, 179-211. Retrieved from <a href="http://synapse.cs.byu.edu/~dan/678/papers/Recurrent/Elman.pdf" rel="nofollow">http://synapse.cs.byu.edu/~dan/678/papers/Recurrent/Elman.pdf</a></p>

CC BY-SA 3.0
2012-01-26T03:49:37.327Z
Point taken; I've gone over this problem enough in my head that I was unaware how many terms I left unspecified. I'll clarify my question shortly and add a diagram of the intended network architecture. On a side note, this seems to serve more as a comment than an answer.
zergylord
CC BY-SA 3.0
2012-01-26T04:04:26.050Z
half comment, half answer-- yes, i can forsee problems arising from that architecture. but moreover, it just didn't fit in the comment box.
Jeff
CC BY-SA 3.0
2012-01-26T04:38:24.530Z
I made a few corrections in order to clear up ambiguities. To be clear, both proposed learning mechanisms are predicting something, but predicting the next state (a matrix of pixels in this case) doesn't require TD since the correct answer is immediately visible in the next time step.
zergylord
CC BY-SA 3.0
2012-01-26T14:55:50.880Z
Are you currently using a RNN or feedforward? A RNN should learn to associate trajectories of the ball with a specific state of the paddle. Sorry if this doesn't help...
Jeff
CC BY-SA 3.0
2012-01-26T03:33:23.840Z
2012-01-26T03:33:23.840Z
Jeff
<p>I've been working on modeling some phenomena involving real-time control in an environment with inherent rewards (specifically, playing a 'pong'-like game), and it's increasingly looking like reinforcement learning by itself won't cut it computationally (I'm currently using a <a href="http://www.stanford.edu/group/pdplab/pdphandbook/handbookch10.html#x26-1290009" rel="nofollow">temporal difference back propagation neural network</a>).</p>

<p>One possible supplemental learning mechanism is to also have the model predict the environment's future state, from which it can learn from in a supervised manner using standard feed forward back propagation.</p>

<p>My current thinking on synthesizing these mechanisms is to have the input layer feed into a hidden layer, which in turn feeds into <strong>both</strong> a reward predicting layer and a separate state predicting layer. In training this net, I simply change the weights via reinforcement learning first and then change them again to account for the state prediction error via back prop.</p>

<p><strong>So my question is this</strong>: Are there any problems you can foresee arising from this architecture? Additionally, has this combination of learning mechanisms been done before, and if so was it done in a similar way?       </p>

CC BY-SA 3.0
2014-09-12T15:04:13.957Z
Do you consider work based on the [Neurological Engineering Framework](http://compneuro.uwaterloo.ca/research/nef.html) as being connectionist?
Seanny123
CC BY-SA 3.0
2012-05-14T17:30:08.457Z
If you are still pursuing this question, [CS.SE](http://cs.stackexchange.com/) might be a great place to ask a follow up question.
Artem Kaznatcheev
CC BY-SA 3.0
2012-06-28T17:49:54.867Z
CLARION has both
http://www.cogsci.rpi.edu/~rsun/clarion.html
vdv
<cognitive-modeling><neural-network><connectionism><artificial-intelligence><reinforcement-learning>
Are there any connectionist models that integrate reinforcement and  fully supervised learning?
zergylord
# Are there any connectionist models that integrate reinforcement and  fully supervised learning?

(Asked by: username_0 on 2012-01-19)

I've been working on modeling some phenomena involving real-time control in an environment with inherent rewards (specifically, playing a 'pong'-like game), and it's increasingly looking like reinforcement learning by itself won't cut it computationally (I'm currently using a [temporal difference back propagation neural network](http://www.stanford.edu/group/pdplab/pdphandbook/handbookch10.html#x26-1290009)).

One possible supplemental learning mechanism is to also have the model predict the environment's future state, from which it can learn from in a supervised manner using standard feed forward back propagation.

My current thinking on synthesizing these mechanisms is to have the input layer feed into a hidden layer, which in turn feeds into **both** a reward predicting layer and a separate state predicting layer. In training this net, I simply change the weights via reinforcement learning first and then change them again to account for the state prediction error via back prop.

**So my question is this**: Are there any problems you can foresee arising from this architecture? Additionally, has this combination of learning mechanisms been done before, and if so was it done in a similar way?

**Tags:** cognitive-modeling, neural-network, connectionism, artificial-intelligence, reinforcement-learning

> **Comments:**
> **username_2**: If you are still pursuing this question, [CS.SE](http://cs.stackexchange.com/) might be a great place to ask a follow up question.  
> **username_3**: CLARION has both http://www.cogsci.rpi.edu/~rsun/clarion.html  
> **username_1**: Do you consider work based on the [Neurological Engineering Framework](http://compneuro.uwaterloo.ca/research/nef.html) as being connectionist?  

---

## 1 answers

### Answer by username_4 on 2012-01-26. Score: 3

I'm not sure I fully understand your design; perhaps you can clarify what you want your network to learn, why TD-learning "isn't cutting it", and what you mean by 'reinforcement' and 'prediction' learning. In particular, TD-learning *is* a reinforcement learning model, and it *does* reward based on predicted (and not just observed) outcomes. However, you seem to be describing reinforcement and prediction learning as orthogonal models, so again, I'm not sure I understand correctly.

As a general suggestion, you might consider using an Elman/Jordan network (e.g. recurrent neural network/RNN). Rather than relying on knowledge of only the current state to make a prediction about the next state, RNN can learn to recognize sequences of events. This is especially useful for predicting future states in a task that unfolds over time (e.g. [1]). I suggest this mostly because you say your task is a 'real-time control' task, but without more knowledge of your task I don't really know if this is appropriate.

As for your suggestion of using two different learning mechanisms to modify a single set of weights-- I don't have the answer, but it seems counterintuitive to me. You're using two different optimization techniques on a single set of data. If the techniques disagree, your network will probably never learn its connection weights. If they agree (i.e., converge on the same answer) then I'm not certain you're adding any value by having two learning mechanisms.

[1] Elman (1990). Finding structure in time. Cognitive Science, 14, 179-211. Retrieved from <http://synapse.cs.byu.edu/~dan/678/papers/Recurrent/Elman.pdf>

> **Comments:**
> **username_0**: Point taken; I've gone over this problem enough in my head that I was unaware how many terms I left unspecified. I'll clarify my question shortly and add a diagram of the intended network architecture. On a side note, this seems to serve more as a comment than an answer.  
> **username_4**: half comment, half answer-- yes, i can forsee problems arising from that architecture. but moreover, it just didn't fit in the comment box.  
> **username_0**: I made a few corrections in order to clear up ambiguities. To be clear, both proposed learning mechanisms are predicting something, but predicting the next state (a matrix of pixels in this case) doesn't require TD since the correct answer is immediately visible in the next time step.  
> **username_4**: Are you currently using a RNN or feedforward? A RNN should learn to associate trajectories of the ball with a specific state of the paddle. Sorry if this doesn't help...  

---

