<p>Indeed, one would like to construct features that make classes linearly separable. However, what patterns exactly should be linearly separable?</p>

<p>It is easy to make patterns in training set linearly separable by greatly increasing feature space dimensionality. However, this will not necessarily imply good generalization (correct classification of new patterns).  Quite opposite, it can lead to overfitting. Thus, it is problematic to use the criterion of linear separability directly to select good nonlinear features for classification. </p>

<p>Deep learning networks adopt different approach. They try to select features with good generalization of patterns irrelative to the classification task. Thus, “useful” nonlinear features are selected without danger of overfitting. Indeed, it doesn’t necessarily ensure linear separability of classes, but it makes sense from induction point of view. If one extract statistically independent factors from initial features, in which these factors are mixed, it will be easier to solve any further classification task.</p>

CC BY-SA 3.0
2014-02-10T19:58:38.020Z
I like your viewpoint about getting statistically independent factors. I'm curious if you have any intuition as to *how* the threshold gates in neural networks try to select features with good generalization, or statistically independent features, or both. That is, why is it that learning nested linear-threshold gates is an *a priori* good idea to achieve these goals?
Joshua Grochow
CC BY-SA 3.0
2014-02-11T09:00:47.703Z
Well, I don't think that this idea is really good. Indeed, only linear features are learned on each level. Then, some fixed nonlinearity is added, and linear features are learned again. It cannot be guaranteed that this fixed nonlinearity will be appropriate to derive independent hidden factors. It is clear from the theory of universal induction based on algorithmic probability. Simpler motivation, why deep learning networks are not so good, is mentioned here

http://videolectures.net/nips2010_tenenbaum_hgm/
Necro0x0Der
CC BY-SA 3.0
2014-02-09T09:01:25.777Z
2014-02-09T09:01:25.777Z
Necro0x0Der
<p>Since each node in a neural network is a weighted linear threshold function, you can in fact view neural networks as mapping into a feature space where it then applies a linear separator, as follows. Consider the functions $g_1,\dotsc,g_k$ computed by the nodes just before the final output node. Here I'm assuming it's a binary classification task, so there's just one output node, but the general case is similar. Then the function $(x_1,\dotsc,x_n) \to (g_1(\vec{x}), \dotsc, g_k(\vec{x}))$ is a mapping into a feature space where the data is then linearly separable (by the final threshold gate).</p>

<p>Why is this useful, compared to the ways kernels are used in SVMs? Well, first of all, I think the features computed by neural networks are more general than the kernels people often use in practice, which come from a limited set of forms. </p>

<p>Second and more importantly, my guess would be that the reason it's useful is that it gives a principled, automatic way to generate the features. In the case of SVMs (if I understand correctly), you more or less have to guess a kernel based on your intuition, prior experience, trial and error, and luck. In the case of neural networks viewed as above, training the neural network builds the features for you.</p>

<p>As you said, the other nodes in the deep neural network - those that come before the penultimate layer of nodes $g_1,\dotsc,g_k$ - are not learning features that make the data linearly separable. I think their primary utility is in giving an automated way to represent and build the features $g_i$ that <em>will</em> make the data linearly separable.</p>

CC BY-SA 3.0
2014-02-09T22:34:00.687Z
2014-02-09T22:34:00.687Z
Joshua Grochow
<p>I thought about this question once, and I convinced myself with an argument similar to this:</p>

<p>Unlike the SVM (kernel), the features are not projected to a different space, instead they are weighted! The weight of the features make them acts like like they would under "gravity" and balances the "space", that way the features that have more weights associated to them actually has more "dominance" over the output (of the activation function). </p>

<p>I like to think of it this way- consider a non linear data set with just 2 attributes, now, after the training phase, each attribute has some weight assigned to them. these weights inherently change their "role in the activation function" thus might actually make one of them to shrink in in size relative to the other, now as this happens if you then again look at the points, you might see that the new data does not look as non linear but instead you can actually make a lot of sense of it.. </p>

<p>This was a very high level "reasoning" with a lot of "handwaving". But intuitively on a high level this is what I think is happening under the hood. </p>

CC BY-SA 3.0
2014-02-08T17:13:46.360Z
I am not an expert in the area, but I don't think weights alone explain the real meat (so to speak) of deep neural networks. Changing weights still gives you a *linear* separator. I think the key has to be in the fact that by combining the outputs of (linear) threshold gates into another level of gates, you can get nonlinear separators, because a threshold function isn't linear as a function (even though it does correspond to a linear separator).
Joshua Grochow
CC BY-SA 3.0
2014-02-09T07:10:46.337Z
@JoshuaGrochow but this doesn't explain how it's different from an SVM, which also gives a nonlinear separator
Suresh Venkat
CC BY-SA 3.0
2014-02-09T15:11:50.027Z
@SureshVenkat: Very true.
Joshua Grochow
CC BY-SA 3.0
2014-02-08T13:00:18.737Z
2014-02-08T13:00:18.737Z
Subhayan
<p>Unless I'm mistaken, deep neural networks are good for learning functions that are nonlinear in the input.</p>

<p>In such cases, the input set is linearly inseparable, so the optimisation problem that results from the approximation problem is not convex, so it cannot be globally optimised with local optimization. Support Vector Machines (try to) get around this by choosing features such that the projection of the input space into the feature space is linearly separable, and we have a convex optimisation problem once again. </p>

<p>I don't suppose that a deep neural network always learns features that make the input set linearly separable in feature space. So what's the point in it learning features?</p>

CC BY-SA 3.0
2014-02-07T15:56:29.063Z
can you rephrase that last question "whats the point in it learning features?" it seems not a question or to not be what you are intending to ask. plausibly the whole _point/raison d'etre_ of ML is to learn _features_. also, ANNs have been learning nonlinear functions ever since early days (1980s), that in particular is not a twist of deep learning. maybe what this question is getting at is the following: deep learning also seems to learn corresponding _deep features_ that cannot be extracted from more "shallow" networks. that is demonstrated in improved performance over the shallow networks.
vzn
<optimization><ne.neural-evol><convex-optimization>
Neural Networks: what's the point of learning features that don't linearly separate?
Alex Dalyac
# Neural Networks: what's the point of learning features that don't linearly separate?

(Asked by: username_0 on 2014-02-07)

Unless I'm mistaken, deep neural networks are good for learning functions that are nonlinear in the input.

In such cases, the input set is linearly inseparable, so the optimisation problem that results from the approximation problem is not convex, so it cannot be globally optimised with local optimization. Support Vector Machines (try to) get around this by choosing features such that the projection of the input space into the feature space is linearly separable, and we have a convex optimisation problem once again.

I don't suppose that a deep neural network always learns features that make the input set linearly separable in feature space. So what's the point in it learning features?

**Tags:** optimization, ne.neural-evol, convex-optimization
> **Comments:**
> **username_1**: can you rephrase that last question "whats the point in it learning features?" it seems not a question or to not be what you are intending to ask. plausibly the whole _point/raison d'etre_ of ML is to learn _features_. also, ANNs have been learning nonlinear functions ever since early days (1980s), that in particular is not a twist of deep learning. maybe what this question is getting at is the following: deep learning also seems to learn corresponding _deep features_ that cannot be extracted from more "shallow" networks. that is demonstrated in improved performance over the shallow networks.  

---

## 3 answers

### Answer by username_2 on 2014-02-09. Score: 1

Indeed, one would like to construct features that make classes linearly separable. However, what patterns exactly should be linearly separable?

It is easy to make patterns in training set linearly separable by greatly increasing feature space dimensionality. However, this will not necessarily imply good generalization (correct classification of new patterns). Quite opposite, it can lead to overfitting. Thus, it is problematic to use the criterion of linear separability directly to select good nonlinear features for classification.

Deep learning networks adopt different approach. They try to select features with good generalization of patterns irrelative to the classification task. Thus, “useful” nonlinear features are selected without danger of overfitting. Indeed, it doesn’t necessarily ensure linear separability of classes, but it makes sense from induction point of view. If one extract statistically independent factors from initial features, in which these factors are mixed, it will be easier to solve any further classification task.

> **Comments:**
> **username_3**: I like your viewpoint about getting statistically independent factors. I'm curious if you have any intuition as to *how* the threshold gates in neural networks try to select features with good generalization, or statistically independent features, or both. That is, why is it that learning nested linear-threshold gates is an *a priori* good idea to achieve these goals?  
> **username_2**: Well, I don't think that this idea is really good. Indeed, only linear features are learned on each level. Then, some fixed nonlinearity is added, and linear features are learned again. It cannot be guaranteed that this fixed nonlinearity will be appropriate to derive independent hidden factors. It is clear from the theory of universal induction based on algorithmic probability. Simpler motivation, why deep learning networks are not so good, is mentioned here  http://videolectures.net/nips2010_tenenbaum_hgm/  

---

### Answer by username_3 on 2014-02-09. Score: 1

Since each node in a neural network is a weighted linear threshold function, you can in fact view neural networks as mapping into a feature space where it then applies a linear separator, as follows. Consider the functions $g\_1,\dotsc,g\_k$ computed by the nodes just before the final output node. Here I'm assuming it's a binary classification task, so there's just one output node, but the general case is similar. Then the function $(x\_1,\dotsc,x\_n) \to (g\_1(\vec{x}), \dotsc, g\_k(\vec{x}))$ is a mapping into a feature space where the data is then linearly separable (by the final threshold gate).

Why is this useful, compared to the ways kernels are used in SVMs? Well, first of all, I think the features computed by neural networks are more general than the kernels people often use in practice, which come from a limited set of forms.

Second and more importantly, my guess would be that the reason it's useful is that it gives a principled, automatic way to generate the features. In the case of SVMs (if I understand correctly), you more or less have to guess a kernel based on your intuition, prior experience, trial and error, and luck. In the case of neural networks viewed as above, training the neural network builds the features for you.

As you said, the other nodes in the deep neural network - those that come before the penultimate layer of nodes $g\_1,\dotsc,g\_k$ - are not learning features that make the data linearly separable. I think their primary utility is in giving an automated way to represent and build the features $g\_i$ that *will* make the data linearly separable.


---

### Answer by username_4 on 2014-02-08. Score: 0

I thought about this question once, and I convinced myself with an argument similar to this:

Unlike the SVM (kernel), the features are not projected to a different space, instead they are weighted! The weight of the features make them acts like like they would under "gravity" and balances the "space", that way the features that have more weights associated to them actually has more "dominance" over the output (of the activation function).

I like to think of it this way- consider a non linear data set with just 2 attributes, now, after the training phase, each attribute has some weight assigned to them. these weights inherently change their "role in the activation function" thus might actually make one of them to shrink in in size relative to the other, now as this happens if you then again look at the points, you might see that the new data does not look as non linear but instead you can actually make a lot of sense of it..

This was a very high level "reasoning" with a lot of "handwaving". But intuitively on a high level this is what I think is happening under the hood.

> **Comments:**
> **username_3**: I am not an expert in the area, but I don't think weights alone explain the real meat (so to speak) of deep neural networks. Changing weights still gives you a *linear* separator. I think the key has to be in the fact that by combining the outputs of (linear) threshold gates into another level of gates, you can get nonlinear separators, because a threshold function isn't linear as a function (even though it does correspond to a linear separator).  
> **username_5**: @JoshuaGrochow but this doesn't explain how it's different from an SVM, which also gives a nonlinear separator  
> **username_3**: @SureshVenkat: Very true.  

---

