<p>I don't know of a particular network model for this job (so my answer will be an incomplete one), but I believe that any Hebbian learning based associative memory can easily be structured to simulate neurogenesis. These unsupervised networks are actually nonlinear dynamic systems that can be understood in terms of their phase spaces. Phase space is the realization of the dynamicity of the network in time. It contains (possibly) several stable states, called minima.</p>

<p>Consider an n-node network. When you add the n+1th node and connect it to the rest of the network via a certain number of connections, what you (hypothetically) see is that the phase space is not reshaped entirely; but it changes smoothly. An important thing to clarify (thanks to Artem Kaznatcheev), is that number n should have a considerably big value. Imagine the extreme case, where n is 1; then the entire structure will be gone.</p>

<p>I am not that good at its mathematics, but in principle, this allows neurogenesis to be simulated on this kind of networks that implement dynamic systems. One can develop algorithms that recruits or removes nodes and connections paralled to training.</p>

<p>How to add nodes: I believe that the answer to this particular question does not effect the neurobiological plausibility of the model. We can even assume that the nodes are there, but not used (due to prior unnecessity), and now we need them and recruit them.</p>

<p>But, as an end note, I must say that I have not developed such a such a metwork before, so I am unable to share my experiences here.</p>

CC BY-SA 3.0
2012-02-12T15:41:33.170Z
There is no reason to expect a smooth or continuous change to the phase-profile when you do a discrete operation like adding a neuron. Further, the whole difficulty of the question lies in HOW to add neurons in a biologically reasonable way, which your answer does not address.
Artem Kaznatcheev
CC BY-SA 3.0
2012-02-12T15:52:13.520Z
I will be editing my answer by adding: 1) number n should have a considerably big value. 2) no need to add neurons, just assume that they are already there.
Kemal Taskin
CC BY-SA 3.0
2012-02-12T18:27:42.743Z
"and now we need them and recruit them" the whole point of my question is to answer HOW to recruit them in a biologically plausible manner.
Artem Kaznatcheev
CC BY-SA 3.0
2012-02-12T15:06:15.933Z
2012-02-12T15:57:09.253Z
2012-02-12T15:57:09.253Z
Kemal Taskin
<p>I don't know of any NN algorithms that match your definition entirely, and I have looked for them (previously and recently). Here are some papers that I think are close or in the direction that you are exploring.</p>
<ul>
<li><p><a href="http://www.bio.vu.nl/enf/vanooyen/papers/vanooyen_nat-rev-neurosci_2011.pdf" rel="nofollow noreferrer">Using theoretical models to analyze neural development</a> (review)</p>
</li>
<li><p><a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3233694/" rel="nofollow noreferrer">An Instruction Language for Self-Construction in the Context of Neural Networks</a></p>
</li>
<li><p><a href="http://dl.acm.org/citation.cfm?id=2001780" rel="nofollow noreferrer">Evolved neurogenesis and synaptogenesis for robotic control: the L-brain model</a></p>
</li>
<li><p><a href="http://Modeling%20new%20neuron%20function:%20a%20history%20of%20using%20computational%20neuroscience%20to%20study%20adult%20neurogenesis" rel="nofollow noreferrer">Modeling new neuron function: a history of using computational neuroscience to study adult neurogenesis</a> (review)</p>
</li>
</ul>
<p>The first two links are computational neuroscience papers that discuss relatively complex models of neurogenesis as it relates to neurodevelopment (rather than adult neurogenesis). Some of these models (including the second paper) do not even involve network activity, and, in their current form, likely none of them are capable or suited for solving practical problems. The third paper is probably closest to what you are looking for (but I don't have access), and the fourth is as its title suggests.</p>
<p>Many network models of neurogenesis focus on neurodevelopment. A fairly obvious conclusion seems to be that the reason for that is that neurogenesis' most important biological role is in neurodevelopment. I have thought a lot about using neuroscience to derive NN algorithms that are both biologically plausible and functional, and have considered neurogenesis within that context. My present conclusion is that, outside of <a href="http://en.wikipedia.org/wiki/Neuroevolution" rel="nofollow noreferrer">neuroevolution</a>, neurogenesis is not currently an ideal focus for NN modeling, because its role in learning and computation seems to be mostly limited to a special case (the hippocampus) that is not well understood (despite its obvious importance).</p>
<p>In regards to cascade correlation, I suspect that a similar effect may be achieved in some biological NNs using only synaptogenesis and synaptic plasticity. Basically, if you have very many neurons and new learning is confined to a minimal number of synapses that are subsequently protected from future modification, then the effect might be the same as always confining new learning to newly-added neurons (as in CC-NN). Such a case would be consistent with <a href="http://Rapid%20formation%20and%20selective%20stabilization%20of%20synapses%20for%20enduring%20motor%20memories." rel="nofollow noreferrer">these</a> findings, for example. In such a model, it would not be biologically plausible, and perhaps not desirable, for each neuron to be connected with each neuron in the preceding and following layer, and thus a system for determining the pattern of exploratory connections would be required. To do that, one could draw from neurodevelopment models such as the above (in order to bias the targets of synaptogenesis, to seed initial connections of the network, or both) or, alternatively, try to derive algorithms that approximate observed connectivity patterns of biological networks.</p>

CC BY-SA 4.0
2012-04-16T06:50:55.947Z
2024-07-28T16:31:12.273Z
2024-07-28T16:31:12.273Z
Matt Munson
<p>One of the reasons artificial neural net algorithms like cascade correlation (<a href="http://www.cs.iastate.edu/~honavar/fahlman.pdf" rel="nofollow noreferrer">pdf</a>) have been generating interest is because they start with a minimal topology (just input and output unit) and recruit new hidden units as learning progresses. The draw from cognitive science (especially a lot of developmental psychology work) is an analogy to <a href="http://en.wikipedia.org/wiki/Neurogenesis" rel="nofollow noreferrer">neuorogenesis</a> (in fact, you will see this analogy mentioned in most papers that use CC-NN). </p>

<p>However, the CC algorithm is not meant as a biologically reasonable algorithm (neither the weight update nor unit recruitment rules are local, for instance), So my question is:</p>

<blockquote>
  <p>Are there popular neural network algorithms that start with a minimal topology and recruit new hidden units, and update network weights in a neurobiologically plausible manner?</p>
</blockquote>

<h3>Related questions</h3>

<p><a href="https://cogsci.stackexchange.com/q/318/29">How are newly created neurons recruited into existing networks?</a></p>

CC BY-SA 3.0
2014-10-16T02:28:09.867Z
@ArtemKaznatcheev Did the reading group ever take off?
Seanny123
CC BY-SA 3.0
2014-10-16T17:28:10.953Z
@Seanny123 there is no reading group on-going for this, and it isn't something that I think about much anymore so it is unlikely that I will start one.
Artem Kaznatcheev
CC BY-SA 3.0
2012-04-16T06:51:54.713Z
I notice that this question was asked a while ago. I would be interested to hear anything you may have come up with since then.
Matt Munson
CC BY-SA 3.0
2012-04-16T09:01:09.203Z
The Wikipedia link has been deleted: http://www.cs.iastate.edu/~honavar/fahlman.pdf. Might get more people answering on CompSci http://cstheory.stackexchange.com/questions/tagged/ne.neural-evol
Chris S
CC BY-SA 3.0
2012-04-16T11:26:32.860Z
@ChrisS unfortunately this not on-topic on cstheory, and I would not want to ask it there. Thanks for catching the dead link!
Artem Kaznatcheev
CC BY-SA 3.0
2012-04-17T13:54:15.357Z
@MattMunson thanks for your answer. I was actually organizing an online reading group on neurogenesis with some colleagues so that we could start seriously thinking about some of these questions. Unfortunately everything got delayed because there were a few other projects ahead in the queue. If you are interested in the reading group then send me an email.
Artem Kaznatcheev
CC BY-SA 3.0
2012-06-06T19:43:55.453Z
FYI your link was broken, I removed it. Feel free to add in a better link if you so choose!
Josh
<developmental-psychology><computational-modeling><neural-network><theoretical-neuroscience><neurogenesis>
Neural networks with biologically plausible accounts of neurogenesis
Artem Kaznatcheev
# Neural networks with biologically plausible accounts of neurogenesis

(Asked by: username_0 on 2012-01-31)

One of the reasons artificial neural net algorithms like cascade correlation ([pdf](http://www.cs.iastate.edu/~honavar/fahlman.pdf)) have been generating interest is because they start with a minimal topology (just input and output unit) and recruit new hidden units as learning progresses. The draw from cognitive science (especially a lot of developmental psychology work) is an analogy to [neuorogenesis](http://en.wikipedia.org/wiki/Neurogenesis) (in fact, you will see this analogy mentioned in most papers that use CC-NN).

However, the CC algorithm is not meant as a biologically reasonable algorithm (neither the weight update nor unit recruitment rules are local, for instance), So my question is:

> Are there popular neural network algorithms that start with a minimal topology and recruit new hidden units, and update network weights in a neurobiologically plausible manner?

### Related questions

[How are newly created neurons recruited into existing networks?](https://cogsci.stackexchange.com/q/318/29)

**Tags:** developmental-psychology, computational-modeling, neural-network, theoretical-neuroscience, neurogenesis

> **Comments:**
> **username_2**: I notice that this question was asked a while ago. I would be interested to hear anything you may have come up with since then.  
> **username_3**: The Wikipedia link has been deleted: http://www.cs.iastate.edu/~honavar/fahlman.pdf. Might get more people answering on CompSci http://cstheory.stackexchange.com/questions/tagged/ne.neural-evol  
> **username_0**: @ChrisS unfortunately this not on-topic on cstheory, and I would not want to ask it there. Thanks for catching the dead link!  
> **username_0**: @MattMunson thanks for your answer. I was actually organizing an online reading group on neurogenesis with some colleagues so that we could start seriously thinking about some of these questions. Unfortunately everything got delayed because there were a few other projects ahead in the queue. If you are interested in the reading group then send me an email.  
> **username_4**: FYI your link was broken, I removed it. Feel free to add in a better link if you so choose!  
> **username_1**: @ArtemKaznatcheev Did the reading group ever take off?  
> **username_0**: @username_1 there is no reading group on-going for this, and it isn't something that I think about much anymore so it is unlikely that I will start one.  

---

## 2 answers

### Answer by username_2 on 2012-04-16. Score: 14 | Accepted answer

I don't know of any NN algorithms that match your definition entirely, and I have looked for them (previously and recently). Here are some papers that I think are close or in the direction that you are exploring.

* [Using theoretical models to analyze neural development](http://www.bio.vu.nl/enf/vanooyen/papers/vanooyen_nat-rev-neurosci_2011.pdf) (review)
* [An Instruction Language for Self-Construction in the Context of Neural Networks](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3233694/)
* [Evolved neurogenesis and synaptogenesis for robotic control: the L-brain model](http://dl.acm.org/citation.cfm?id=2001780)
* [Modeling new neuron function: a history of using computational neuroscience to study adult neurogenesis](http://Modeling%20new%20neuron%20function:%20a%20history%20of%20using%20computational%20neuroscience%20to%20study%20adult%20neurogenesis) (review)

The first two links are computational neuroscience papers that discuss relatively complex models of neurogenesis as it relates to neurodevelopment (rather than adult neurogenesis). Some of these models (including the second paper) do not even involve network activity, and, in their current form, likely none of them are capable or suited for solving practical problems. The third paper is probably closest to what you are looking for (but I don't have access), and the fourth is as its title suggests.

Many network models of neurogenesis focus on neurodevelopment. A fairly obvious conclusion seems to be that the reason for that is that neurogenesis' most important biological role is in neurodevelopment. I have thought a lot about using neuroscience to derive NN algorithms that are both biologically plausible and functional, and have considered neurogenesis within that context. My present conclusion is that, outside of [neuroevolution](http://en.wikipedia.org/wiki/Neuroevolution), neurogenesis is not currently an ideal focus for NN modeling, because its role in learning and computation seems to be mostly limited to a special case (the hippocampus) that is not well understood (despite its obvious importance).

In regards to cascade correlation, I suspect that a similar effect may be achieved in some biological NNs using only synaptogenesis and synaptic plasticity. Basically, if you have very many neurons and new learning is confined to a minimal number of synapses that are subsequently protected from future modification, then the effect might be the same as always confining new learning to newly-added neurons (as in CC-NN). Such a case would be consistent with [these](http://Rapid%20formation%20and%20selective%20stabilization%20of%20synapses%20for%20enduring%20motor%20memories.) findings, for example. In such a model, it would not be biologically plausible, and perhaps not desirable, for each neuron to be connected with each neuron in the preceding and following layer, and thus a system for determining the pattern of exploratory connections would be required. To do that, one could draw from neurodevelopment models such as the above (in order to bias the targets of synaptogenesis, to seed initial connections of the network, or both) or, alternatively, try to derive algorithms that approximate observed connectivity patterns of biological networks.


---

### Answer by username_5 on 2012-02-12. Score: 1

I don't know of a particular network model for this job (so my answer will be an incomplete one), but I believe that any Hebbian learning based associative memory can easily be structured to simulate neurogenesis. These unsupervised networks are actually nonlinear dynamic systems that can be understood in terms of their phase spaces. Phase space is the realization of the dynamicity of the network in time. It contains (possibly) several stable states, called minima.

Consider an n-node network. When you add the n+1th node and connect it to the rest of the network via a certain number of connections, what you (hypothetically) see is that the phase space is not reshaped entirely; but it changes smoothly. An important thing to clarify (thanks to username_0), is that number n should have a considerably big value. Imagine the extreme case, where n is 1; then the entire structure will be gone.

I am not that good at its mathematics, but in principle, this allows neurogenesis to be simulated on this kind of networks that implement dynamic systems. One can develop algorithms that recruits or removes nodes and connections paralled to training.

How to add nodes: I believe that the answer to this particular question does not effect the neurobiological plausibility of the model. We can even assume that the nodes are there, but not used (due to prior unnecessity), and now we need them and recruit them.

But, as an end note, I must say that I have not developed such a such a metwork before, so I am unable to share my experiences here.

> **Comments:**
> **username_0**: There is no reason to expect a smooth or continuous change to the phase-profile when you do a discrete operation like adding a neuron. Further, the whole difficulty of the question lies in HOW to add neurons in a biologically reasonable way, which your answer does not address.  
> **username_5**: I will be editing my answer by adding: 1) number n should have a considerably big value. 2) no need to add neurons, just assume that they are already there.  
> **username_0**: "and now we need them and recruit them" the whole point of my question is to answer HOW to recruit them in a biologically plausible manner.  

---

