  Supervised methods for object delineation in remote sensing require labeled
ground-truth data. Gathering sufficient high quality ground-truth data is
difficult, especially when targets are of irregular shape or difficult to
distinguish from background or neighboring objects. Tree crown delineation
provides key information from remote sensing images for forestry, ecology, and
management. However, tree crowns in remote sensing imagery are often difficult
to label and annotate due to irregular shape, overlapping canopies, shadowing,
and indistinct edges. There are also multiple approaches to annotation in this
field (e.g., rectangular boxes vs. convex polygons) that further contribute to
annotation imprecision. However, current evaluation methods do not account for
this uncertainty in annotations, and quantitative metrics for evaluation can
vary across multiple annotators. In this paper, we address these limitations by
developing an adaptation of the Rand index for weakly-labeled crown delineation
that we call RandCrowns. Our new RandCrowns evaluation metric provides a method
to appropriately evaluate delineated tree crowns while taking into account
imprecision in the ground-truth delineations. The RandCrowns metric
reformulates the Rand index by adjusting the areas over which each term of the
index is computed to account for uncertain and imprecise object delineation
labels. Quantitative comparisons to the commonly used intersection over union
method shows a decrease in the variance generated by differences among multiple
annotators. Combined with qualitative examples, our results suggest that the
RandCrowns metric is more robust for scoring target delineations in the
presence of uncertainty and imprecision in annotations that are inherent to
tree crown delineation.
