  It has recently been conjectured that neural network solution sets reachable
via stochastic gradient descent (SGD) are convex, considering permutation
invariances (Entezari et al., 2022). This means that a linear path can connect
two independent solutions with low loss, given the weights of one of the models
are appropriately permuted. However, current methods to test this theory often
require very wide networks to succeed. In this work, we conjecture that more
generally, the SGD solution set is a "star domain" that contains a "star model"
that is linearly connected to all the other solutions via paths with low loss
values, modulo permutations. We propose the Starlight algorithm that finds a
star model of a given learning task. We validate our claim by showing that this
star model is linearly connected with other independently found solutions. As
an additional benefit of our study, we demonstrate better uncertainty estimates
on the Bayesian Model Averaging over the obtained star domain. Further, we
demonstrate star models as potential substitutes for model ensembles. Our code
is available at https://github.com/aktsonthalia/starlight.
