  We present multi-point optimization: an optimization technique that allows to
train several models simultaneously without the need to keep the parameters of
each one individually. The proposed method is used for a thorough empirical
analysis of the loss landscape of neural networks. By extensive experiments on
FashionMNIST and CIFAR10 datasets we demonstrate two things: 1) loss surface is
surprisingly diverse and intricate in terms of landscape patterns it contains,
and 2) adding batch normalization makes it more smooth. Source code to
reproduce all the reported results is available on GitHub:
https://github.com/universome/loss-patterns.
