<p>Backpropagation is computationally expensive. Has any research been done on a partially greedy implementation of it?</p>

<p>Intuition: at the beginning of training, big rough learning steps can be taken, so do we really need to tune each individual weight? </p>

<p>If we were to plot time series of every parameter as its value changes over training time, might we get correlations between certain parameters, particularly at the beginning of training?</p>

<p>For example, since the gradient propagates backwards, we might get correlations between the parameters of neurons behind one another. Such correlations could suggest that it's a waste of time to compute partial derivatives for all parameters, i.e we can use the partial derivative for a neuron in front to approximate the partial derivatives for the neurons behind it?</p>

<p>Has anyone seen any articles exploring anything like this?</p>

CC BY-SA 3.0
2014-04-26T14:20:19.413Z
afaik backpropagation is basically built out of gradient descent which is a greedy algorithm. roughly all nonzero update patterns must be temporary as the network converges. most analysis is done on the converged networks. however, new/"more intelligent" update rules are continually being invented & are continually under research... a newer trend seems to be building larger networks out of converged smaller networks...
vzn
CC BY-SA 3.0
2014-04-26T14:29:53.960Z
so if we were to plot time series of every parameter as its value changes over training time, you don't think we'd get any correlations between certain parameters, particularly at the beginning of training?
Could you provide link(s) to paper(s) about building larger networks from converged smaller networks?
Alex Dalyac
CC BY-SA 3.0
2014-04-26T14:38:39.583Z
it would be highly problem dependent to look at weight trends during convergence, agree that it is worth some investigation & there could be some insight there wrt better update rules, but generally ANNs are treated as black boxes until _after_ convergence. also in final convergence the uniqueness of all the weights is generally theoretically important (except smaller weights ofc). re building larger networks out of converged smaller networks, it seems to show up in newer deep learning papers, will post ref if run across one (its more another question...)
vzn
CC BY-SA 3.0
2014-04-27T01:07:29.987Z
ok found this, it seems a relatively newer technique. this paper [Unsupervised Learning of Hierarchical Representations with Convolutional Deep
Belief Networks](http://web.eecs.umich.edu/~honglak/cacm2011-researchHighlights-convDBN.pdf) by Honglak Lee describes (eg sec2.2) freezing lower level/layer network weights after training/convergence and then subsequently using them as inputs for training higher level layers.
vzn
CC BY-SA 3.0
2014-04-27T11:08:43.943Z
thanks for the heads up! If networks such as [OverFeat](https://github.com/sermanet/OverFeat) are successfully being imported for tasks in which the data is believed to be similar (eg in this case, image classification), then maybe the problem dependency is not as high as we'd think?
Alex Dalyac
<ne.neural-evol>
Greedy Backpropagation: has anyone researched it?
Alex Dalyac
# Greedy Backpropagation: has anyone researched it?

(Asked by: username_0 on 2014-04-24)

Backpropagation is computationally expensive. Has any research been done on a partially greedy implementation of it?

Intuition: at the beginning of training, big rough learning steps can be taken, so do we really need to tune each individual weight?

If we were to plot time series of every parameter as its value changes over training time, might we get correlations between certain parameters, particularly at the beginning of training?

For example, since the gradient propagates backwards, we might get correlations between the parameters of neurons behind one another. Such correlations could suggest that it's a waste of time to compute partial derivatives for all parameters, i.e we can use the partial derivative for a neuron in front to approximate the partial derivatives for the neurons behind it?

Has anyone seen any articles exploring anything like this?

**Tags:** ne.neural-evol
> **Comments:**
> **username_1**: afaik backpropagation is basically built out of gradient descent which is a greedy algorithm. roughly all nonzero update patterns must be temporary as the network converges. most analysis is done on the converged networks. however, new/"more intelligent" update rules are continually being invented & are continually under research... a newer trend seems to be building larger networks out of converged smaller networks...  
> **username_0**: so if we were to plot time series of every parameter as its value changes over training time, you don't think we'd get any correlations between certain parameters, particularly at the beginning of training? Could you provide link(s) to paper(s) about building larger networks from converged smaller networks?  
> **username_1**: it would be highly problem dependent to look at weight trends during convergence, agree that it is worth some investigation & there could be some insight there wrt better update rules, but generally ANNs are treated as black boxes until _after_ convergence. also in final convergence the uniqueness of all the weights is generally theoretically important (except smaller weights ofc). re building larger networks out of converged smaller networks, it seems to show up in newer deep learning papers, will post ref if run across one (its more another question...)  
> **username_1**: ok found this, it seems a relatively newer technique. this paper [Unsupervised Learning of Hierarchical Representations with Convolutional Deep Belief Networks](http://web.eecs.umich.edu/~honglak/cacm2011-researchHighlights-convDBN.pdf) by Honglak Lee describes (eg sec2.2) freezing lower level/layer network weights after training/convergence and then subsequently using them as inputs for training higher level layers.  
> **username_0**: thanks for the heads up! If networks such as [OverFeat](https://github.com/sermanet/OverFeat) are successfully being imported for tasks in which the data is believed to be similar (eg in this case, image classification), then maybe the problem dependency is not as high as we'd think?  

---

## 0 answers

