  Research on neural networks has focused on understanding a single model
trained on a single dataset. However, relatively little is known about the
relationships between different models, particularly those trained or tested on
different datasets. We address this by studying how the weight space and the
underlying loss landscape of different models are interconnected.
  Specifically, we demonstrate that finetuned models that were optimized for
high performance, reside in well-defined regions in weight space, and vice
versa -- that any model that resides anywhere in those regions also exhibits
high performance. Notably, we show that language models that have been
finetuned on the same dataset form a tight cluster in the weight space, while
models finetuned on different datasets from the same underlying task form a
looser cluster. Moreover, traversing around the region between the models leads
to new models that perform comparably or even better than models obtained via
finetuning, even on tasks that the original models were not finetuned on.
  Our findings provide insight into the relationships between models,
demonstrating that a model positioned between two similar models can acquire
the knowledge of both. We leverage this and design a method for selecting a
better model for efficient finetuning. Specifically, we show that starting from
the center of the region is as effective, if not more, than using the
pretrained model in 11 out of 12 datasets, resulting in an average accuracy
improvement of 3.06.
