  Artificial intelligence (AI)-based methods are showing substantial promise in
segmenting oncologic positron emission tomography (PET) images. For clinical
translation of these methods, assessing their performance on clinically
relevant tasks is important. However, these methods are typically evaluated
using metrics that may not correlate with the task performance. One such widely
used metric is the Dice score, a figure of merit that measures the spatial
overlap between the estimated segmentation and a reference standard (e.g.,
manual segmentation). In this work, we investigated whether evaluating AI-based
segmentation methods using Dice scores yields a similar interpretation as
evaluation on the clinical tasks of quantifying metabolic tumor volume (MTV)
and total lesion glycolysis (TLG) of primary tumor from PET images of patients
with non-small cell lung cancer. The investigation was conducted via a
retrospective analysis with the ECOG-ACRIN 6668/RTOG 0235 multi-center clinical
trial data. Specifically, we evaluated different structures of a commonly used
AI-based segmentation method using both Dice scores and the accuracy in
quantifying MTV/TLG. Our results show that evaluation using Dice scores can
lead to findings that are inconsistent with evaluation using the task-based
figure of merit. Thus, our study motivates the need for objective task-based
evaluation of AI-based segmentation methods for quantitative PET.
