<p><strong>Short answer:</strong> Not necessarily. Likely nothing fishy is going on.</p>

<p><strong>Longer answer:</strong>
The Universal Approximation Theorem (UAT) says nothing about an individual network's capacity to approximate a function. Moreover it says nothing about trainability or generalization.</p>

<p>The UAT roughly says that for any $\epsilon&gt;0$ and any continuous real-valued function $F$ on compact domain $\Omega$ there exists a feed-forward network $f$ with a single hidden layer (with appropriate weights) such that $| F(x)-f(x)|&lt;\epsilon$ for all $x\in\Omega$. Inductively using the shallow UAT, one can prove a deep UAT. So in <strong>theory</strong> depth offers no more <strong>expressiveness</strong>. </p>

<p>Note that this merely guarantees the existence of <strong>some</strong> appropriately tuned neural network, and not that any particular network can be trained up to that desired accuracy. </p>

<p>On the other hand, deep neural networks have been seen to be often much better in many applications. This is often attributed to their ability to learn hierarchical representations of the data. The UAT says nothing about generalization, which is the bar by which success is usually measured in practice. <a href="https://arxiv.org/pdf/1603.00988.pdf" rel="nofollow noreferrer">Here</a> it is argued that deep networks can approximate compositional functions with less parameters than shallow counterparts. </p>

<p>But on the other-other hand, it was noticed that sometimes adding more depth can hurt when training a network. But why should it? One could in principle just set the new layer to be the identity. Apparently the later layers were actually trying to approximate the identity but doing it very poorly. So why not learn how different your layer is from the identity? This observation led to <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">Residual Learning</a> which in turn led to some really, <em>really</em> deep networks. </p>

CC BY-SA 3.0
2018-03-21T19:20:24.267Z
2018-03-21T19:20:24.267Z
Christian Bueno
<p>In case you're still looking for more information on this, I'll chip in my two cents. </p>

<p>It may help to think of neural networks as just fitting some function based on the training data. Each hidden layer increases the ability of the neural network to fit more complex functions. A network without hidden layers, often called a perceptron, is only able to fit linear (technically linearly separable) functions. As a rule of thumb, each hidden layer adds the ability to fit one higher degree polynomial function. So for example a network with one hidden layer should be able to fit a quadratic function, two hidden layers can fit a cubic function and so on.</p>

<p>Perhaps contrary to your intuition, the cosine you're trying to train is actually a complex dataset for a neural network, precisely because it fits polynomial functions. Implicitly the network will relax to weights that are in principle approximating your cosine with a polynomial. If you look at <a href="http://en.wikipedia.org/wiki/Taylor_series#Approximation_and_convergence" rel="nofollow">the tailor expansion</a> for a sine for example, you'll see that a close approximation would require a 7th degree polynomial, which together with the rather small $\frac{1}{7 !}$ weight should prove quite a challenge for your network to train.</p>

<p>It might also be interesting to note that the approximation only holds well for $−1 &lt; x &lt; 1$, since a typical polynomial will quickly diverge from the sine/cosine beyond this interval. You might want to take that into account for your training set.</p>

<p>To wrap up it up, the primary reason you should be cautious with just adding hidden layers is the problem of overfitting. If you add too many layers (ie more layers than would be required for the -possibly unknown- polynomial underlying your training data), the network will typically fit too tightly to your training dataset, making it less useful for interpolation or extrapolation, or just evaluating new data outside the set. </p>

<p>A good example would actually be that <a href="http://en.wikipedia.org/wiki/Taylor_series#Approximation_and_convergence" rel="nofollow">tailor expansion</a> again. If you read on in that section you'll come across the note that higher degree polynomials only marginally improve the accuracy in $−1 &lt; x &lt; 1$, while they perform (much) worse outside this interval. I should note that this may not exactly be the same problem, but it does illustrate what could go wrong.</p>

CC BY-SA 3.0
2015-04-13T10:53:15.880Z
I don't understand how adding a hidden layer gives the ability to fit one higher degree of a polynomial function, given a sufficient number of hidden nodes shouldn't just one hidden layer be enough to fit any continuous function? Multiple hidden layers would probably be able to perform better since you are adding an extra composition of the activation functions (I'm not absolutely certain of this) but I don't see how they're directly linked with the approximation of higher degree polynomials.
mesllo
CC BY-SA 3.0
2015-04-17T13:58:15.007Z
I didn't mean to imply there actually is a direct mathematical link, in fact I'd be hard-pressed to rigorously argue why this should be so myself. I wouldn't even presume to do any hand-waving argument to linear algebraic curve fitting. As I said though, it's a rule of thumb which in my experience has proven to be sound.
Fasermaler
CC BY-SA 3.0
2015-03-13T23:27:40.410Z
2015-03-13T23:50:07.587Z
2015-03-13T23:50:07.587Z
Fasermaler
<p>If I have a multi hidden-layered neural network that is getting a better approximation for a function than a single one, does that mean that there is something "fishy" about my multi layered one because of the Universal Approximation Theorem?</p>

<p>I am assuming here that all weights are pre-initialized to 1 (I have my reasons) and they are trained on the same exact dataset.</p>

CC BY-SA 3.0
2015-03-12T12:26:03.463Z
Shouldn't multi-layered NN work better if you have enough training data? What are your concerns?
R B
CC BY-SA 3.0
2015-03-12T13:17:40.837Z
I may have misread the UAT, but as far as I know, more layers should NOT improve the network performance which is why I was concerned in the first place. Unfortunately I have misplaced the source of this insight.
mesllo
CC BY-SA 3.0
2015-03-12T13:25:44.660Z
I have found the source: "To summarize this concept, start without a hidden
layer and examine the results. Then add one hidden layer and
determine which network provides the best results, Adding
additional layers should not improve network capabilities but
may if the data contain complex patterns."

This is taken from An Evalutation Of Artificial Neural Network Modelling For Manpower Analysis by Brian James Byrne. For the dataset, I am merely approximating a number of cosine functions and couldn't think how complex such a dataset could be as it only contains a range of real values from -1 to 1
mesllo
CC BY-SA 3.0
2015-03-12T20:06:26.590Z
How many internal nodes are you allowing in your single hidden layer network? With enough hidden nodes, it can compute (essentially) anything that's computable.
Aryeh
CC BY-SA 3.0
2015-03-12T20:57:15.553Z
I am testing 2 layered networks having 2 neurons each and then compare these with a single layered network having 2 neurons (for example). Using for example 50 neurons and a single layer, yes it computes it fine, I am exploring multi-layered networks though
mesllo
CC BY-SA 3.0
2015-03-14T19:03:36.843Z
I feel this might be more suitable for [stats.se].
Kaveh
<machine-learning><ne.neural-evol><universal-computation>
Are single hidden-layered neural networks at least as good as multi hidden-layered neural networks?
mesllo
# Are single hidden-layered neural networks at least as good as multi hidden-layered neural networks?

(Asked by: username_0 on 2015-03-12)

If I have a multi hidden-layered neural network that is getting a better approximation for a function than a single one, does that mean that there is something "fishy" about my multi layered one because of the Universal Approximation Theorem?

I am assuming here that all weights are pre-initialized to 1 (I have my reasons) and they are trained on the same exact dataset.

**Tags:** machine-learning, ne.neural-evol, universal-computation
> **Comments:**
> **username_1**: Shouldn't multi-layered NN work better if you have enough training data? What are your concerns?  
> **username_0**: I may have misread the UAT, but as far as I know, more layers should NOT improve the network performance which is why I was concerned in the first place. Unfortunately I have misplaced the source of this insight.  
> **username_0**: I have found the source: "To summarize this concept, start without a hidden layer and examine the results. Then add one hidden layer and determine which network provides the best results, Adding additional layers should not improve network capabilities but may if the data contain complex patterns."  This is taken from An Evalutation Of Artificial Neural Network Modelling For Manpower Analysis by Brian James Byrne. For the dataset, I am merely approximating a number of cosine functions and couldn't think how complex such a dataset could be as it only contains a range of real values from -1 to 1  
> **username_2**: How many internal nodes are you allowing in your single hidden layer network? With enough hidden nodes, it can compute (essentially) anything that's computable.  
> **username_0**: I am testing 2 layered networks having 2 neurons each and then compare these with a single layered network having 2 neurons (for example). Using for example 50 neurons and a single layer, yes it computes it fine, I am exploring multi-layered networks though  
> **username_3**: I feel this might be more suitable for [stats.se].  

---

## 2 answers

### Answer by username_4 on 2018-03-21. Score: 2 | Accepted answer

**Short answer:** Not necessarily. Likely nothing fishy is going on.

**Longer answer:**
The Universal Approximation Theorem (UAT) says nothing about an individual network's capacity to approximate a function. Moreover it says nothing about trainability or generalization.

The UAT roughly says that for any $\epsilon>0$ and any continuous real-valued function $F$ on compact domain $\Omega$ there exists a feed-forward network $f$ with a single hidden layer (with appropriate weights) such that $| F(x)-f(x)|<\epsilon$ for all $x\in\Omega$. Inductively using the shallow UAT, one can prove a deep UAT. So in **theory** depth offers no more **expressiveness**.

Note that this merely guarantees the existence of **some** appropriately tuned neural network, and not that any particular network can be trained up to that desired accuracy.

On the other hand, deep neural networks have been seen to be often much better in many applications. This is often attributed to their ability to learn hierarchical representations of the data. The UAT says nothing about generalization, which is the bar by which success is usually measured in practice. [Here](https://arxiv.org/pdf/1603.00988.pdf) it is argued that deep networks can approximate compositional functions with less parameters than shallow counterparts.

But on the other-other hand, it was noticed that sometimes adding more depth can hurt when training a network. But why should it? One could in principle just set the new layer to be the identity. Apparently the later layers were actually trying to approximate the identity but doing it very poorly. So why not learn how different your layer is from the identity? This observation led to [Residual Learning](https://arxiv.org/pdf/1512.03385.pdf) which in turn led to some really, *really* deep networks.


---

### Answer by username_5 on 2015-03-13. Score: 1

In case you're still looking for more information on this, I'll chip in my two cents.

It may help to think of neural networks as just fitting some function based on the training data. Each hidden layer increases the ability of the neural network to fit more complex functions. A network without hidden layers, often called a perceptron, is only able to fit linear (technically linearly separable) functions. As a rule of thumb, each hidden layer adds the ability to fit one higher degree polynomial function. So for example a network with one hidden layer should be able to fit a quadratic function, two hidden layers can fit a cubic function and so on.

Perhaps contrary to your intuition, the cosine you're trying to train is actually a complex dataset for a neural network, precisely because it fits polynomial functions. Implicitly the network will relax to weights that are in principle approximating your cosine with a polynomial. If you look at [the tailor expansion](http://en.wikipedia.org/wiki/Taylor_series#Approximation_and_convergence) for a sine for example, you'll see that a close approximation would require a 7th degree polynomial, which together with the rather small $\frac{1}{7 !}$ weight should prove quite a challenge for your network to train.

It might also be interesting to note that the approximation only holds well for $−1 < x < 1$, since a typical polynomial will quickly diverge from the sine/cosine beyond this interval. You might want to take that into account for your training set.

To wrap up it up, the primary reason you should be cautious with just adding hidden layers is the problem of overfitting. If you add too many layers (ie more layers than would be required for the -possibly unknown- polynomial underlying your training data), the network will typically fit too tightly to your training dataset, making it less useful for interpolation or extrapolation, or just evaluating new data outside the set.

A good example would actually be that [tailor expansion](http://en.wikipedia.org/wiki/Taylor_series#Approximation_and_convergence) again. If you read on in that section you'll come across the note that higher degree polynomials only marginally improve the accuracy in $−1 < x < 1$, while they perform (much) worse outside this interval. I should note that this may not exactly be the same problem, but it does illustrate what could go wrong.

> **Comments:**
> **username_0**: I don't understand how adding a hidden layer gives the ability to fit one higher degree of a polynomial function, given a sufficient number of hidden nodes shouldn't just one hidden layer be enough to fit any continuous function? Multiple hidden layers would probably be able to perform better since you are adding an extra composition of the activation functions (I'm not absolutely certain of this) but I don't see how they're directly linked with the approximation of higher degree polynomials.  
> **username_5**: I didn't mean to imply there actually is a direct mathematical link, in fact I'd be hard-pressed to rigorously argue why this should be so myself. I wouldn't even presume to do any hand-waving argument to linear algebraic curve fitting. As I said though, it's a rule of thumb which in my experience has proven to be sound.  

---

