  Short-term forecasting of residential and commercial building energy
consumption is widely used in power systems and continues to grow in
importance. Data-driven short-term load forecasting (STLF), although promising,
has suffered from a lack of open, large-scale datasets with high building
diversity. This has hindered exploring the pretrain-then-fine-tune paradigm for
STLF. To help address this, we present BuildingsBench, which consists of: 1)
Buildings-900K, a large-scale dataset of 900K simulated buildings representing
the U.S. building stock; and 2) an evaluation platform with over 1,900 real
residential and commercial buildings from 7 open datasets. BuildingsBench
benchmarks two under-explored tasks: zero-shot STLF, where a pretrained model
is evaluated on unseen buildings without fine-tuning, and transfer learning,
where a pretrained model is fine-tuned on a target building. The main finding
of our benchmark analysis is that synthetically pretrained models generalize
surprisingly well to real commercial buildings. An exploration of the effect of
increasing dataset size and diversity on zero-shot commercial building
performance reveals a power-law with diminishing returns. We also show that
fine-tuning pretrained models on real commercial and residential buildings
improves performance for a majority of target buildings. We hope that
BuildingsBench encourages and facilitates future research on generalizable
STLF. All datasets and code can be accessed from
https://github.com/NREL/BuildingsBench.
