  Research is needed to explore the limitations and potential for improvement
of machine learning for building energy prediction. With this aim, the ASHRAE
Great Energy Predictor III (GEPIII) Kaggle competition was launched in 2019.
This effort was the largest building energy meter machine learning competition
of its kind, with 4,370 participants who submitted 39,403 predictions. The test
data set included two years of hourly whole building readings from 2,380 meters
in 1,448 buildings at 16 locations. This paper analyzes the various sources and
types of residual model error from an aggregation of the competition's top 50
solutions. This analysis reveals the limitations for machine learning using the
standard model inputs of historical meter, weather, and basic building
metadata. The errors are classified according to timeframe, behavior,
magnitude, and incidence in single buildings or across a campus. The results
show machine learning models have errors within a range of acceptability
(RMSLE_scaled =< 0.1) on 79.1% of the test data. Lower magnitude (in-range)
model errors (0.1 < RMSLE_scaled =< 0.3) occur in 16.1% of the test data. These
errors could be remedied using innovative training data from onsite and
web-based sources. Higher magnitude (out-of-range) errors (RMSLE_scaled > 0.3)
occur in 4.8% of the test data and are unlikely to be accurately predicted.
