  Interactive segmentation has recently been explored to effectively and
efficiently harvest high-quality segmentation masks by iteratively
incorporating user hints. While iterative in nature, most existing interactive
segmentation methods tend to ignore the dynamics of successive interactions and
take each interaction independently. We here propose to model iterative
interactive image segmentation with a Markov decision process (MDP) and solve
it with reinforcement learning (RL) where each voxel is treated as an agent.
Considering the large exploration space for voxel-wise prediction and the
dependence among neighboring voxels for the segmentation tasks, multi-agent
reinforcement learning is adopted, where the voxel-level policy is shared among
agents. Considering that boundary voxels are more important for segmentation,
we further introduce a boundary-aware reward, which consists of a global reward
in the form of relative cross-entropy gain, to update the policy in a
constrained direction, and a boundary reward in the form of relative weight, to
emphasize the correctness of boundary predictions. To combine the advantages of
different types of interactions, i.e., simple and efficient for point-clicking,
and stable and robust for scribbles, we propose a supervoxel-clicking based
interaction design. Experimental results on four benchmark datasets have shown
that the proposed method significantly outperforms the state-of-the-arts, with
the advantage of fewer interactions, higher accuracy, and enhanced robustness.
