  Deep learning has been successful in automating the design of features in
machine learning pipelines. However, the algorithms optimizing neural network
parameters remain largely hand-designed and computationally inefficient. We
study if we can use deep learning to directly predict these parameters by
exploiting the past knowledge of training other networks. We introduce a
large-scale dataset of diverse computational graphs of neural architectures -
DeepNets-1M - and use it to explore parameter prediction on CIFAR-10 and
ImageNet. By leveraging advances in graph neural networks, we propose a
hypernetwork that can predict performant parameters in a single forward pass
taking a fraction of a second, even on a CPU. The proposed model achieves
surprisingly good performance on unseen and diverse networks. For example, it
is able to predict all 24 million parameters of a ResNet-50 achieving a 60%
accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks
approaches 50%. Our task along with the model and results can potentially lead
to a new, more computationally efficient paradigm of training networks. Our
model also learns a strong representation of neural architectures enabling
their analysis.
