  As a fundamental and extensively studied task in computer vision, image
segmentation aims to locate and identify different semantic concepts at the
pixel level. Recently, inspired by In-Context Learning (ICL), several
generalist segmentation frameworks have been proposed, providing a promising
paradigm for segmenting specific objects. However, existing works mostly ignore
the value of visual prompts or simply apply similarity sorting to select
contextual examples. In this paper, we focus on rethinking and improving the
example selection strategy. By comprehensive comparisons, we first demonstrate
that ICL-based segmentation models are sensitive to different contexts.
Furthermore, empirical evidence indicates that the diversity of contextual
prompts plays a crucial role in guiding segmentation. Based on the above
insights, we propose a new stepwise context search method. Different from
previous works, we construct a small yet rich candidate pool and adaptively
search the well-matched contexts. More importantly, this method effectively
reduces the annotation cost by compacting the search space. Extensive
experiments show that our method is an effective strategy for selecting
examples and enhancing segmentation performance.
