  In recent years, artificial neural networks have achieved state-of-the-art
performance for predicting the responses of neurons in the visual cortex to
natural stimuli. However, they require a time consuming parameter optimization
process for accurately modeling the tuning function of newly observed neurons,
which prohibits many applications including real-time, closed-loop experiments.
We overcome this limitation by formulating the problem as $K$-shot prediction
to directly infer a neuron's tuning function from a small set of
stimulus-response pairs using a Neural Process. This required us to developed a
Factorized Neural Process, which embeds the observed set into a latent space
partitioned into the receptive field location and the tuning function
properties. We show on simulated responses that the predictions and
reconstructed receptive fields from the Factorized Neural Process approach
ground truth with increasing number of trials. Critically, the latent
representation that summarizes the tuning function of a neuron is inferred in a
quick, single forward pass through the network. Finally, we validate this
approach on real neural data from visual cortex and find that the predictive
accuracy is comparable to -- and for small $K$ even greater than --
optimization based approaches, while being substantially faster. We believe
this novel deep learning systems identification framework will facilitate
better real-time integration of artificial neural network modeling into
neuroscience experiments.
