  Graph neural networks have become increasingly popular in recent years due to
their ability to naturally encode relational input data and their ability to
scale to large graphs by operating on a sparse representation of graph
adjacency matrices. As we look to scale up these models using custom hardware,
a natural assumption would be that we need hardware tailored to sparse
operations and/or dynamic control flow. In this work, we question this
assumption by scaling up sparse graph neural networks using a platform targeted
at dense computation on fixed-size data. Drawing inspiration from optimization
of numerical algorithms on sparse matrices, we develop techniques that enable
training the sparse graph neural network model from Allamanis et al. [2018] in
13 minutes using a 512-core TPUv2 Pod, whereas the original training takes
almost a day.
