  Current methods to determine the energy efficiency of buildings require
on-site visits of certified energy auditors which makes the process slow,
costly, and geographically incomplete. To accelerate the identification of
promising retrofit targets on a large scale, we propose to estimate building
energy efficiency from widely available and remotely sensed data sources only,
namely street view, aerial view, footprint, and satellite-borne land surface
temperature (LST) data. After collecting data for almost 40,000 buildings in
the United Kingdom, we combine these data sources by training multiple
end-to-end deep learning models with the objective to classify buildings as
energy efficient (EU rating A-D) or inefficient (EU rating E-G). After
evaluating the trained models quantitatively as well as qualitatively, we
extend our analysis by studying the predictive power of each data source in an
ablation study. We find that the end-to-end deep learning model trained on all
four data sources achieves a macro-averaged F1 score of 64.64% and outperforms
the k-NN and SVM-based baseline models by 14.13 to 12.02 percentage points,
respectively. Thus, this work shows the potential and complementary nature of
remotely sensed data in predicting energy efficiency and opens up new
opportunities for future work to integrate additional data sources.
