  Neural networks that process the parameters of other neural networks find
applications in domains as diverse as classifying implicit neural
representations, generating neural network weights, and predicting
generalization errors. However, existing approaches either overlook the
inherent permutation symmetry in the neural network or rely on intricate
weight-sharing patterns to achieve equivariance, while ignoring the impact of
the network architecture itself. In this work, we propose to represent neural
networks as computational graphs of parameters, which allows us to harness
powerful graph neural networks and transformers that preserve permutation
symmetry. Consequently, our approach enables a single model to encode neural
computational graphs with diverse architectures. We showcase the effectiveness
of our method on a wide range of tasks, including classification and editing of
implicit neural representations, predicting generalization performance, and
learning to optimize, while consistently outperforming state-of-the-art
methods. The source code is open-sourced at
https://github.com/mkofinas/neural-graphs.
