  Training neural networks involves finding minima of a high-dimensional
non-convex loss function. Knowledge of the structure of this energy landscape
is sparse. Relaxing from linear interpolations, we construct continuous paths
between minima of recent neural network architectures on CIFAR10 and CIFAR100.
Surprisingly, the paths are essentially flat in both the training and test
landscapes. This implies that neural networks have enough capacity for
structural changes, or that these changes are small between minima. Also, each
minimum has at least one vanishing Hessian eigenvalue in addition to those
resulting from trivial invariance.
