  We propose the Taylorformer for time series and other random processes. Its
two key components are: 1) the LocalTaylor wrapper to learn how and when to use
Taylor series-based approximations for predictions, and 2) the MHA-X attention
block which makes predictions in a way inspired by how Gaussian Processes' mean
predictions are linear smoothings of contextual data. Taylorformer outperforms
the state-of-the-art on several forecasting datasets, including electricity,
oil temperatures and exchange rates with at least 14% improvement in MSE on all
tasks, and better likelihood on 5/6 classic Neural Process tasks such as
meta-learning 1D functions. Taylorformer combines desirable features from the
Neural Process (uncertainty-aware predictions and consistency) and forecasting
(predictive accuracy) literature, two previously distinct bodies.
