<p>This predates the start of the "deep learning craze" in the mid-2000s, but for me the cascade correlation algorithm (Fahlman &amp; Lebiere, 1989; <a href="http://repository.cmu.edu/cgi/viewcontent.cgi?article=2937&amp;context=compsci&amp;sei-redir=1&amp;referer=http%3A%2F%2Fscholar.google.ca%2Fscholar%3Fq%3Dcascade-correlation%2Balgorithm%26hl%3Den%26as_sdt%3D0%26as_vis%3D1%26oi%3Dscholart%26sa%3DX%26ei%3DpA_zUqyuKZLUsATusIDgDA%26ved%3D0CC0QgQMwAA#search=%22cascade-correlation%20algorithm%22" rel="nofollow noreferrer">pdf</a>) is <em>the</em> build-your-own-topology NN algorithm. I am not sure how popular the algorithm is in ML now-a-days, but it is still popular in cognitive science <a href="https://cogsci.stackexchange.com/q/214/29">despite it's unrealistic grounding in biology</a>. If you want intersections with genetic algorithms (like the one @vzn points out with more recent examples) then that has been done with CC as well (Potter, 1992; <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.135.7792&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">pdf</a>).</p>

<p>You might also find the following questions on other SEs to be of interest:</p>

<ul>
<li><a href="https://cs.stackexchange.com/q/455/55">Why do neural networks seem to perform better with restrictions placed on their topology?</a></li>
<li><a href="https://stats.stackexchange.com/q/33967/4872">Dynamically adjusting NN architecture: inventing the unnecessary?</a> (also <a href="http://metaoptimize.com/qa/questions/10799/dynamically-adjusting-nn-architecture-inventing-the-unnecessary" rel="nofollow noreferrer">on MetaOptimize</a>)</li>
</ul>

CC BY-SA 3.0
2014-02-06T04:54:53.223Z
2014-02-06T04:54:53.223Z
2017-04-13T12:56:17.630Z
Artem Kaznatcheev
<p>this is generally an open question under active research; however there is quite a bit of research using <a href="http://en.wikipedia.org/wiki/Genetic_algorithm" rel="nofollow">GAs</a> to determine NN architectures see eg</p>

<ul>
<li><p><a href="http://www.iidia.com.ar/rgm/articulos/RCSJ-27-15-24.pdf" rel="nofollow">Finding Optimal Neural Network Architecture Using Genetic Algorithms</a> Britos et al</p></li>
<li><p><a href="http://www.sfu.ca/~rgencay/jarticles/pa-genetic.pdf" rel="nofollow">Using genetic algorithms to select architecture of a feedforward artificial neural network</a> Arifovic/Gencay</p></li>
</ul>

<p>however <em>caveat</em> this type of approach is very expensive computationally and possibly hasnt yet been used with <a href="http://en.wikipedia.org/wiki/Deep_learning" rel="nofollow">deep learning</a>.</p>

CC BY-SA 3.0
2014-02-01T20:55:55.123Z
2014-02-01T20:55:55.123Z
vzn
<p>here are two other more recent papers that touch on this. as AK cites, cascade correlation algorithms were designed around ~1990 for this purpose of dynamic ANN growth but they seem not to be applied in deep learning experiments so far. the 2nd paper below cites cascade correlation as an inspiration for an incremental/online algorithm that adds feature detectors over time built out of lower-level feature detectors.</p>

<ul>
<li><p><a href="http://www.geminoid.jp/~minato/papers/BenAmor06a.pdf" rel="nofollow">Learning Android Control using Growing Neural Networks</a> 2006 Amor et al</p>

<blockquote>
  <p>Fixed sized neural networks are a popular technique for learning the adaptive control of non-linear plants. When applied to the complex control of android robots, however, they suffer from serious limitations, such as the moving target problem i.e. the interference between old and newly learned knowledge. To overcome these problems, we propose the use of growing neural networks in a new learning framework based on the process of consolidation. </p>
</blockquote></li>
<li><p>this paper comes from <a href="https://sites.google.com/site/deeplearningnips2012/posters" rel="nofollow">Deep learning and unsupervised feature learning NIPS workshop 2012</a>. <a href="http://www.eng.uwaterloo.ca/~jbergstr/files/nips_dl_2012/Paper%2019.pdf" rel="nofollow">Online Representation Search and
Its Interactions with Unsupervised Learning</a> by Mahmood/Sutton. see fig 4 p4. the network performance vs incremental growth in feature detectors is graphed.</p>

<blockquote>
  <p>We consider the problem of finding good hidden units, or features, for use in multilayer neural networks. Solution methods that generate candidate features, evaluate them, and retain the most useful ones (such as cascade correlation and NEAT), we call representation search methods. In this paper, we explore novel representation search methods in an online setting, compare them with two simple unsupervised learning algorithms that also scale online.</p>
</blockquote></li>
</ul>

CC BY-SA 3.0
2014-02-07T03:05:44.943Z
2014-02-07T03:05:44.943Z
vzn
<p>Yoshua Benhgio's <a href="http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf" rel="nofollow">Learning Deep Architectures for AI</a> book mentions that </p>

<blockquote>
  <p>we should [...] strive to develop learning algorithms that use the data to determine the depth of the ﬁnal architecture.</p>
</blockquote>

<p>Would anyone know of any algorithms proposed thus far to achieve this?</p>

<hr>

<p>So far I have come across:</p>

<p>The <a href="http://iopscience.iop.org/0305-4470/22/12/019" rel="nofollow">tiling algorithm</a> for building a feed-forward network to learn a Boolean function. It adds layers as well as units, but Boolean functions aren't too relevant for applied problems.</p>

CC BY-SA 3.0
2014-02-06T16:10:41.277Z
an emerging idea is to somehow measure how the response of how "random" neurons are in response to inputs, and one builds new neural connections on top of neurons that are "nonrandom" ie "orderly" and that find nonrandom patterns...
vzn
CC BY-SA 3.0
2014-02-07T00:34:38.333Z
@vzn what do you mean by "measure how the response of how "random" neurons are in response to inputs", and what do you mean by "neural connections on top of neurons"? would you have a paper to link us to? sounds very interesting!
Alex Dalyac
CC BY-SA 3.0
2014-02-07T00:37:54.987Z
the basic idea is that neurons act as feature detectors. neurons succeed or fail as feature detectors. if the output of a neuron is random, the neuron has failed to find a feature. features are nonrandom/extracted order. new features are hierarchically built out of other features. havent seen this in the literature so far. the closest concept comes from [neural darwinism](http://en.wikipedia.org/wiki/Neural_Darwinism) originated by edelman but apparently not yet incorporated into ("artificial"/algorithmic) connectionist models. may look for more papers on the subj wrt application in ML.
vzn
<machine-learning><ne.neural-evol>
Which algorithms have been proposed to learn the architecture of a deep neural network?
Alex Dalyac
# Which algorithms have been proposed to learn the architecture of a deep neural network?

(Asked by: username_0 on 2014-02-01)

Yoshua Benhgio's [Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf) book mentions that

> we should [...] strive to develop learning algorithms that use the data to determine the depth of the ﬁnal architecture.

Would anyone know of any algorithms proposed thus far to achieve this?

---

So far I have come across:

The [tiling algorithm](http://iopscience.iop.org/0305-4470/22/12/019) for building a feed-forward network to learn a Boolean function. It adds layers as well as units, but Boolean functions aren't too relevant for applied problems.

**Tags:** machine-learning, ne.neural-evol

> **Comments:**
> **username_1**: an emerging idea is to somehow measure how the response of how "random" neurons are in response to inputs, and one builds new neural connections on top of neurons that are "nonrandom" ie "orderly" and that find nonrandom patterns...  
> **username_0**: @username_1 what do you mean by "measure how the response of how "random" neurons are in response to inputs", and what do you mean by "neural connections on top of neurons"? would you have a paper to link us to? sounds very interesting!  
> **username_1**: the basic idea is that neurons act as feature detectors. neurons succeed or fail as feature detectors. if the output of a neuron is random, the neuron has failed to find a feature. features are nonrandom/extracted order. new features are hierarchically built out of other features. havent seen this in the literature so far. the closest concept comes from [neural darwinism](http://en.wikipedia.org/wiki/Neural_Darwinism) originated by edelman but apparently not yet incorporated into ("artificial"/algorithmic) connectionist models. may look for more papers on the subj wrt application in ML.  

---

## 3 answers

### Answer by username_2 on 2014-02-06. Score: 3

This predates the start of the "deep learning craze" in the mid-2000s, but for me the cascade correlation algorithm (Fahlman & Lebiere, 1989; [pdf](http://repository.cmu.edu/cgi/viewcontent.cgi?article=2937&context=compsci&sei-redir=1&referer=http%3A%2F%2Fscholar.google.ca%2Fscholar%3Fq%3Dcascade-correlation%2Balgorithm%26hl%3Den%26as_sdt%3D0%26as_vis%3D1%26oi%3Dscholart%26sa%3DX%26ei%3DpA_zUqyuKZLUsATusIDgDA%26ved%3D0CC0QgQMwAA#search=%22cascade-correlation%20algorithm%22)) is *the* build-your-own-topology NN algorithm. I am not sure how popular the algorithm is in ML now-a-days, but it is still popular in cognitive science [despite it's unrealistic grounding in biology](https://cogsci.stackexchange.com/q/214/29). If you want intersections with genetic algorithms (like the one @username_1 points out with more recent examples) then that has been done with CC as well (Potter, 1992; [pdf](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.135.7792&rep=rep1&type=pdf)).

You might also find the following questions on other SEs to be of interest:

* [Why do neural networks seem to perform better with restrictions placed on their topology?](https://cs.stackexchange.com/q/455/55)
* [Dynamically adjusting NN architecture: inventing the unnecessary?](https://stats.stackexchange.com/q/33967/4872) (also [on MetaOptimize](http://metaoptimize.com/qa/questions/10799/dynamically-adjusting-nn-architecture-inventing-the-unnecessary))


---

### Answer by username_1 on 2014-02-01. Score: 0

this is generally an open question under active research; however there is quite a bit of research using [GAs](http://en.wikipedia.org/wiki/Genetic_algorithm) to determine NN architectures see eg

* [Finding Optimal Neural Network Architecture Using Genetic Algorithms](http://www.iidia.com.ar/rgm/articulos/RCSJ-27-15-24.pdf) Britos et al
* [Using genetic algorithms to select architecture of a feedforward artificial neural network](http://www.sfu.ca/~rgencay/jarticles/pa-genetic.pdf) Arifovic/Gencay

however *caveat* this type of approach is very expensive computationally and possibly hasnt yet been used with [deep learning](http://en.wikipedia.org/wiki/Deep_learning).


---

### Answer by username_1 on 2014-02-07. Score: -1

here are two other more recent papers that touch on this. as AK cites, cascade correlation algorithms were designed around ~1990 for this purpose of dynamic ANN growth but they seem not to be applied in deep learning experiments so far. the 2nd paper below cites cascade correlation as an inspiration for an incremental/online algorithm that adds feature detectors over time built out of lower-level feature detectors.

* [Learning Android Control using Growing Neural Networks](http://www.geminoid.jp/~minato/papers/BenAmor06a.pdf) 2006 Amor et al

  > Fixed sized neural networks are a popular technique for learning the adaptive control of non-linear plants. When applied to the complex control of android robots, however, they suffer from serious limitations, such as the moving target problem i.e. the interference between old and newly learned knowledge. To overcome these problems, we propose the use of growing neural networks in a new learning framework based on the process of consolidation.
* this paper comes from [Deep learning and unsupervised feature learning NIPS workshop 2012](https://sites.google.com/site/deeplearningnips2012/posters). [Online Representation Search and
  Its Interactions with Unsupervised Learning](http://www.eng.uwaterloo.ca/~jbergstr/files/nips_dl_2012/Paper%2019.pdf) by Mahmood/Sutton. see fig 4 p4. the network performance vs incremental growth in feature detectors is graphed.

  > We consider the problem of finding good hidden units, or features, for use in multilayer neural networks. Solution methods that generate candidate features, evaluate them, and retain the most useful ones (such as cascade correlation and NEAT), we call representation search methods. In this paper, we explore novel representation search methods in an online setting, compare them with two simple unsupervised learning algorithms that also scale online.


---

