  Self-Supervised Learning (SSL) has been shown to learn useful and
information-preserving representations. Neural Networks (NNs) are widely
applied, yet their weight space is still not fully understood. Therefore, we
propose to use SSL to learn hyper-representations of the weights of populations
of NNs. To that end, we introduce domain specific data augmentations and an
adapted attention architecture. Our empirical evaluation demonstrates that
self-supervised representation learning in this domain is able to recover
diverse NN model characteristics. Further, we show that the proposed learned
representations outperform prior work for predicting hyper-parameters, test
accuracy, and generalization gap as well as transfer to out-of-distribution
settings.
