<p>If the singular values of each layer are smaller than 1, then you can have vanishing gradients.  If the singular values are larger than 1, you can have exploding gradients.  Both cause problems for training.</p>

<p>Why?  As a heuristic, imagine that all the singular values for each layer were 2, and you have <span class="math-container">$n$</span> layers.  Then the singular values for the entire network would be <span class="math-container">$2^n$</span>.  So, if the singular values of the layers are larger than 1, the singular values for the entire network can grow exponentially in the number of layers.  Similarly if they're less than 1, they can vanish exponentially fast.</p>

CC BY-SA 4.0
2019-11-08T19:40:16.690Z
2019-11-08T19:40:16.690Z
D.W.
<p>I'm reading the paper (<a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf</a>) from Glorot and Bengio. There is something that I don't understand at the abstract section on page 1.</p>

<p>"Training may be more difficult when the singular values of the Jacobian associated with each layer are far from 1"</p>

<p>Why is the singular value of the Jacobian important for training?</p>

<p>Why is the singular value being far from 1 difficult for training?</p>

<machine-learning>
Singular Value in Machine Learning
user55240
# Singular Value in Machine Learning

(Asked by: username_0 on 2019-11-08)

I'm reading the paper (<http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf>) from Glorot and Bengio. There is something that I don't understand at the abstract section on page 1.

"Training may be more difficult when the singular values of the Jacobian associated with each layer are far from 1"

Why is the singular value of the Jacobian important for training?

Why is the singular value being far from 1 difficult for training?

**Tags:** machine-learning

---

## 1 answers

### Answer by username_1 on 2019-11-08. Score: 1

If the singular values of each layer are smaller than 1, then you can have vanishing gradients. If the singular values are larger than 1, you can have exploding gradients. Both cause problems for training.

Why? As a heuristic, imagine that all the singular values for each layer were 2, and you have $n$ layers. Then the singular values for the entire network would be $2^n$. So, if the singular values of the layers are larger than 1, the singular values for the entire network can grow exponentially in the number of layers. Similarly if they're less than 1, they can vanish exponentially fast.


---

