<p>I was trying to understand if there was a way to express the back-propagation equations from neural networks in a better way as to understand them better. I believe the equations can be written in a recursive manner as follows (using the notation from <a href="https://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf" rel="nofollow">these notes</a> from Andrew Ng):</p>

<p>$$ \delta^{(n_l)}_{i} = \left( \sum^{s_{l+1}}_{j=1} W^{(l)}_{ji} \delta^{(l+1)}_{j} \right) f'(z^{(l)}_{i})$$</p>

<p>I know that some recursions/recurrences have closed form "equations"/solutions. A typical example would be the Fibonacci sequence (or things that use Akra-Bazzi to solve them). I was wondering, because of the similarity to recursion of this equation, does something like that exist for back-propagation equations in Neural Networks? If there is not, does that mean its impossible to get such an equations? I was wondering if there were formal mathematical proofs arguments that might explain the impossibility of such an attempt or any work that has been tried to that goal.</p>

<p>In reality, it seems that the main use of this equation is algorithmically. But I was hoping that having some sort of equation (or even approximation to this recurrence) could help understand back-propagation and neural networks better. Might this require new mathematics to achieve something like this? Or maybe trying to do something like this makes no sense? If so why so?</p>

CC BY-SA 3.0
2015-08-14T00:30:47.200Z
I think you will have a better luck asking this on [stats.se]. You can flag the question for moderator attention if you want the question to be migrated there.
Kaveh
CC BY-SA 3.0
2015-08-14T02:08:02.387Z
Well, in the case that f is the identity (or just linear), and assuming that each weight matrix is the same $W$, then we have $\delta^{(n)} = (W^{m - n})^T$ where $T$ denotes transpose, and $m$ denotes the layer number of the output layer. If $W$ is diagonalizable with $W = R^T D R$ with diagonal matrix $D$ and orthogonal matrices $R$, then you have a closed form given by $\delta^{(n)} = R^T D^{m-n} R$. Hopefully this helps with your intuition a bit, but note that in practice it's much much more efficient to just to the multiplications, since finding eigenvalues is in generally much harder.
SorcererofDM
CC BY-SA 3.0
2015-08-14T02:12:32.117Z
I think nobody's trying to solve for a "close form" here because this is about as simple as it gets, and any abstraction on top is entirely unnecessary from a practical point of view. With the field moving as it is, there are plenty of unformalized mathematics.
SorcererofDM
CC BY-SA 3.0
2015-08-14T03:57:45.827Z
The other tricky part is that solving non-linear equations is *really hard*. This is a big reason why neural networks are so general, and why training them requires so many herustics - it's very difficult to prove things about such complex systems, without solving many open problems in mathematics first. There are proofs you can make about regularization and probability distributions and such, but to actually "solve" neural networks and immediately jump to the optimal form is most likely impossible.
Phylliida
CC BY-SA 3.0
2015-08-15T20:28:58.417Z
@DanielleEnsign do you know of any specific open problems that one would have to be solve before solving this one?
Charlie Parker
CC BY-SA 3.0
2015-09-09T03:37:55.493Z
"Backpropagation, at its core, simply consists of repeatedly applying the chain rule through all of the possible paths in our network." http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4
S Huntsman
<machine-learning><recursion><recursive>
Is there a closed form equation for the back-propagation equation update in Neural Networks?
Charlie Parker
# Is there a closed form equation for the back-propagation equation update in Neural Networks?

(Asked by: username_0 on 2015-08-13)

I was trying to understand if there was a way to express the back-propagation equations from neural networks in a better way as to understand them better. I believe the equations can be written in a recursive manner as follows (using the notation from [these notes](https://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf) from Andrew Ng):

$$ \delta^{(n\_l)}\_{i} = \left( \sum^{s\_{l+1}}\_{j=1} W^{(l)}\_{ji} \delta^{(l+1)}\_{j} \right) f'(z^{(l)}\_{i})$$

I know that some recursions/recurrences have closed form "equations"/solutions. A typical example would be the Fibonacci sequence (or things that use Akra-Bazzi to solve them). I was wondering, because of the similarity to recursion of this equation, does something like that exist for back-propagation equations in Neural Networks? If there is not, does that mean its impossible to get such an equations? I was wondering if there were formal mathematical proofs arguments that might explain the impossibility of such an attempt or any work that has been tried to that goal.

In reality, it seems that the main use of this equation is algorithmically. But I was hoping that having some sort of equation (or even approximation to this recurrence) could help understand back-propagation and neural networks better. Might this require new mathematics to achieve something like this? Or maybe trying to do something like this makes no sense? If so why so?

**Tags:** machine-learning, recursion, recursive
> **Comments:**
> **username_1**: I think you will have a better luck asking this on [stats.se]. You can flag the question for moderator attention if you want the question to be migrated there.  
> **username_2**: Well, in the case that f is the identity (or just linear), and assuming that each weight matrix is the same $W$, then we have $\delta^{(n)} = (W^{m - n})^T$ where $T$ denotes transpose, and $m$ denotes the layer number of the output layer. If $W$ is diagonalizable with $W = R^T D R$ with diagonal matrix $D$ and orthogonal matrices $R$, then you have a closed form given by $\delta^{(n)} = R^T D^{m-n} R$. Hopefully this helps with your intuition a bit, but note that in practice it's much much more efficient to just to the multiplications, since finding eigenvalues is in generally much harder.  
> **username_2**: I think nobody's trying to solve for a "close form" here because this is about as simple as it gets, and any abstraction on top is entirely unnecessary from a practical point of view. With the field moving as it is, there are plenty of unformalized mathematics.  
> **username_3**: The other tricky part is that solving non-linear equations is *really hard*. This is a big reason why neural networks are so general, and why training them requires so many herustics - it's very difficult to prove things about such complex systems, without solving many open problems in mathematics first. There are proofs you can make about regularization and probability distributions and such, but to actually "solve" neural networks and immediately jump to the optimal form is most likely impossible.  
> **username_0**: @DanielleEnsign do you know of any specific open problems that one would have to be solve before solving this one?  
> **username_4**: "Backpropagation, at its core, simply consists of repeatedly applying the chain rule through all of the possible paths in our network." http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4  

---

## 0 answers

