  Interactive Image Segmentation (IIS) has emerged as a promising technique for
decreasing annotation time. Substantial progress has been made in pre- and
post-processing for IIS, but the critical issue of interaction ambiguity,
notably hindering segmentation quality, has been under-researched. To address
this, we introduce AdaptiveClick -- a click-aware transformer incorporating an
adaptive focal loss that tackles annotation inconsistencies with tools for
mask- and pixel-level ambiguity resolution. To the best of our knowledge,
AdaptiveClick is the first transformer-based, mask-adaptive segmentation
framework for IIS. The key ingredient of our method is the Click-Aware
Mask-adaptive transformer Decoder (CAMD), which enhances the interaction
between click and image features. Additionally, AdaptiveClick enables
pixel-adaptive differentiation of hard and easy samples in the decision space,
independent of their varying distributions. This is primarily achieved by
optimizing a generalized Adaptive Focal Loss (AFL) with a theoretical
guarantee, where two adaptive coefficients control the ratio of gradient values
for hard and easy pixels. Our analysis reveals that the commonly used Focal and
BCE losses can be considered special cases of the proposed AFL. With a plain
ViT backbone, extensive experimental results on nine datasets demonstrate the
superiority of AdaptiveClick compared to state-of-the-art methods. The source
code is publicly available at https://github.com/lab206/AdaptiveClick.
