  Current AI training infrastructure is dominated by single instruction
multiple data (SIMD) and systolic array architectures, such as Graphics
Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at
accelerating parallel workloads and dense vector matrix multiplications.
Potentially more efficient neural network models utilizing sparsity and
recurrence cannot leverage the full power of SIMD processor and are thus at a
severe disadvantage compared to today's prominent parallel architectures like
Transformers and CNNs, thereby hindering the path towards more sustainable AI.
To overcome this limitation, we explore sparse and recurrent model training on
a massively parallel multiple instruction multiple data (MIMD) architecture
with distributed local memory. We implement a training routine based on
backpropagation through time (BPTT) for the brain-inspired class of Spiking
Neural Networks (SNNs) that feature binary sparse activations. We observe a
massive advantage in using sparse activation tensors with a MIMD processor, the
Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our
results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x
gains for higher levels of activation sparsity, without a significant slowdown
in training convergence or reduction in final model performance. Furthermore,
our results show highly promising trends for both single and multi IPU
configurations as we scale up to larger model sizes. Our work paves the way
towards more efficient, non-standard models via AI training hardware beyond
GPUs, and competitive large scale SNN models.
