  State-of-the-art systems neuroscience experiments yield large-scale
multimodal data, and these data sets require new tools for analysis. Inspired
by the success of large pretrained models in vision and language domains, we
reframe the analysis of large-scale, cellular-resolution neuronal spiking data
into an autoregressive spatiotemporal generation problem. Neuroformer is a
multimodal, multitask generative pretrained transformer (GPT) model that is
specifically designed to handle the intricacies of data in systems
neuroscience. It scales linearly with feature size, can process an arbitrary
number of modalities, and is adaptable to downstream tasks, such as predicting
behavior. We first trained Neuroformer on simulated datasets, and found that it
both accurately predicted simulated neuronal circuit activity, and also
intrinsically inferred the underlying neural circuit connectivity, including
direction. When pretrained to decode neural responses, the model predicted the
behavior of a mouse with only few-shot fine-tuning, suggesting that the model
begins learning how to do so directly from the neural representations
themselves, without any explicit supervision. We used an ablation study to show
that joint training on neuronal responses and behavior boosted performance,
highlighting the model's ability to associate behavioral and neural
representations in an unsupervised manner. These findings show that Neuroformer
can analyze neural datasets and their emergent properties, informing the
development of models and hypotheses associated with the brain.
