  Effective integration planning for small, distributed solar photovoltaic (PV)
arrays into electric power grids requires access to high quality data: the
location and power capacity of individual solar PV arrays. Unfortunately,
national databases of small-scale solar PV do not exist; those that do are
limited in their spatial resolution, typically aggregated up to state or
national levels. While several promising approaches for solar PV detection have
been published, strategies for evaluating the performance of these models are
often highly heterogeneous from study to study. The resulting comparison of
these methods for practical applications for energy assessments becomes
challenging and may imply that the reported performance evaluations are overly
optimistic. The heterogeneity comes in many forms, each of which we explore in
this work: the level of spatial aggregation, the validation of ground truth,
inconsistencies in the training and validation datasets, and the degree of
diversity of the locations and sensors from which the training and validation
data originate. For each, we discuss emerging practices from the literature to
address them or suggest directions of future research. As part of our
investigation, we evaluate solar PV identification performance in two large
regions. Our findings suggest that traditional performance evaluation of the
automated identification of solar PV from satellite imagery may be optimistic
due to common limitations in the validation process. The takeaways from this
work are intended to inform and catalyze the large-scale practical application
of automated solar PV assessment techniques by energy researchers and
professionals.
