  This paper introduces WeatherFormer, a transformer encoder-based model
designed to learn robust weather features from minimal observations. It
addresses the challenge of modeling complex weather dynamics from small
datasets, a bottleneck for many prediction tasks in agriculture, epidemiology,
and climate science. WeatherFormer was pretrained on a large pretraining
dataset comprised of 39 years of satellite measurements across the Americas.
With a novel pretraining task and fine-tuning, WeatherFormer achieves
state-of-the-art performance in county-level soybean yield prediction and
influenza forecasting. Technical innovations include a unique spatiotemporal
encoding that captures geographical, annual, and seasonal variations, adapting
the transformer architecture to continuous weather data, and a pretraining
strategy to learn representations that are robust to missing weather features.
This paper for the first time demonstrates the effectiveness of pretraining
large transformer encoder models for weather-dependent applications across
multiple domains.
