  The second-order properties of the training loss have a massive impact on the
optimization dynamics of deep learning models. Fort & Scherlis (2019)
discovered that a large excess of positive curvature and local convexity of the
loss Hessian is associated with highly trainable initial points located in a
region coined the "Goldilocks zone". Only a handful of subsequent studies
touched upon this relationship, so it remains largely unexplained. In this
paper, we present a rigorous and comprehensive analysis of the Goldilocks zone
for homogeneous neural networks. In particular, we derive the fundamental
condition resulting in excess of positive curvature of the loss, explaining and
refining its conventionally accepted connection to the initialization norm.
Further, we relate the excess of positive curvature to model confidence, low
initial loss, and a previously unknown type of vanishing cross-entropy loss
gradient. To understand the importance of excessive positive curvature for
trainability of deep networks, we optimize fully-connected and convolutional
architectures outside the Goldilocks zone and analyze the emergent behaviors.
We find that strong model performance is not perfectly aligned with the
Goldilocks zone, calling for further research into this relationship.
