  Learning representations of well-trained neural network models holds the
promise to provide an understanding of the inner workings of those models.
However, previous work has either faced limitations when processing larger
networks or was task-specific to either discriminative or generative tasks.
This paper introduces the SANE approach to weight-space learning. SANE
overcomes previous limitations by learning task-agnostic representations of
neural networks that are scalable to larger models of varying architectures and
that show capabilities beyond a single task. Our method extends the idea of
hyper-representations towards sequential processing of subsets of neural
network weights, thus allowing one to embed larger neural networks as a set of
tokens into the learned representation space. SANE reveals global model
information from layer-wise embeddings, and it can sequentially generate unseen
neural network models, which was unattainable with previous
hyper-representation learning methods. Extensive empirical evaluation
demonstrates that SANE matches or exceeds state-of-the-art performance on
several weight representation learning benchmarks, particularly in
initialization for new tasks and larger ResNet architectures.
