  Grounding DINO and the Segment Anything Model (SAM) have achieved impressive
performance in zero-shot object detection and image segmentation, respectively.
Together, they have a great potential to revolutionize applications in
zero-shot semantic segmentation or data annotation. Yet, in specialized domains
like medical image segmentation, objects of interest (e.g., organs, tissues,
and tumors) may not fall in existing class names. To address this problem, the
referring expression comprehension (REC) ability of Grounding DINO is leveraged
to detect arbitrary targets by their language descriptions. However, recent
studies have highlighted severe limitation of the REC framework in this
application setting owing to its tendency to make false positive predictions
when the target is absent in the given image. And, while this bottleneck is
central to the prospect of open-set semantic segmentation, it is still largely
unknown how much improvement can be achieved by studying the prediction errors.
To this end, we perform empirical studies on six publicly available datasets
across different domains and reveal that these errors consistently follow a
predictable pattern and can, thus, be mitigated by a simple strategy.
Specifically, we show that false positive detections with appreciable
confidence scores generally occupy large image areas and can usually be
filtered by their relative sizes. More importantly, we expect these
observations to inspire future research in improving REC-based detection and
automated segmentation. Meanwhile, we evaluate the performance of SAM on
multiple datasets from various specialized domains and report significant
improvements in segmentation performance and annotation time savings over
manual approaches.
