Konstantin Sofiiuk
Ilia A. Petrov
Anton Konushin
Recent works on click-based interactive segmentation have demonstrated state-of-the-art results by using various inference-time optimization schemes. These methods are considerably more computationally expensive compared to feedforward approaches, as they require performing backward passes through a network during inference and are hard to deploy on mobile frameworks that usually support only forward passes. In this paper, we extensively evaluate various design choices for interactive segmentation and discover that new state-of-the-art results can be obtained without any additional optimization schemes. Thus, we propose a simple feedforward model for click-based interactive segmentation that employs the segmentation masks from previous steps. It allows not only to segment an entirely new object, but also to start with an external mask and correct it. When analyzing the performance of models trained on different datasets, we observe that the choice of a training dataset greatly impacts the quality of interactive segmentation. We find that the models trained on a combination of COCO and LVIS with diverse and high-quality annotations show performance superior to all existing models. The code and trained models are available at \url{https://github.com/saic-vul/ritm_interactive_segmentation.
main.tex
%% 
%% Copyright 2019-2020 Elsevier Ltd
%% 
%% This file is part of the 'CAS Bundle'.
%% --------------------------------------
%% 
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%% 
%% The list of all files belonging to the 'CAS Bundle' is
%% given in the file `manifest.txt'.
%% 
%% Template article for cas-sc documentclass for 
%% single column output.

%\documentclass[a4paper,fleqn,longmktitle]{cas-sc}
\documentclass[a4paper,fleqn]{cas-sc}

%%% Custom packages
\usepackage[numbers]{natbib}
\usepackage{csquotes}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{float}

%%% Custom macros
\def\tsc#1{\csdef{#1}{\textsc{\lowercase{#1}}\xspace}}
\tsc{WGM}
\tsc{QE}
\tsc{EP}
\tsc{PMS}
\tsc{BEC}
\tsc{DE}
\def\etal{\emph{et al.}}
\def\eg{\emph{e.g.}}
%%%

\begin{document}
\let\WriteBookmarks\relax
\def\floatpagepagefraction{1}
\def\textpagefraction{.001}
\shorttitle{Reviving Iterative Training with Mask Guidance for Interactive Segmentation}
\shortauthors{K.Sofiiuk, I.Petrov, A.Konushin}
%\begin{frontmatter}

\title [mode = title]{Reviving Iterative Training with Mask Guidance for Interactive Segmentation}

\author{Konstantin Sofiiuk}
\ead{k.sofiiuk@samsung.com}
\address{Visual Understanding lab., AI Center Moscow, Samsung Electronics Co., Lesnaya 5C, Moscow, Russia}


\author{Ilia A. Petrov}[orcid=0000-0002-8900-1071]
\cormark[1]
\ead{ilia.petrov@samsung.com}

\author{Anton Konushin}[orcid=0000-0002-6152-0021]
\cormark[2]
\ead{a.konushin@samsung.com}

\cortext[cor1]{Corresponding author}
\cortext[cor2]{Principal corresponding author}

%%%%%%%%% ---------------------------- ABSTRACT ----------------------------
\begin{abstract}
Recent works on click-based interactive segmentation have demonstrated state-of-the-art results by using various inference-time optimization schemes. These methods are considerably more computationally expensive compared to feedforward approaches, as they require performing backward passes through a network during inference and are hard to deploy on mobile frameworks that usually support only forward passes. In this paper, we extensively evaluate various design choices for interactive segmentation and discover that new state-of-the-art results can be obtained without any additional optimization schemes. Thus, we propose a simple feedforward model for click-based interactive segmentation that employs the segmentation masks from previous steps. It allows not only to segment an entirely new object, but also to start with an external mask and correct it. When analyzing the performance of models trained on different datasets, we observe that the choice of a training dataset greatly impacts the quality of interactive segmentation. We find that the models trained on a combination of COCO and LVIS with diverse and high-quality annotations show performance superior to all existing models. The code and trained models are available at \url{https://github.com/saic-vul/ritm_interactive_segmentation}.
\end{abstract}

%%%%%%%%% ------------------------------ KEYWORDS ------------------------------
\begin{keywords}
interactive segmentation \sep segmentation \sep mask refinement
\end{keywords}

\maketitle

%%%%%%%%% ---------------------------- INTRODUCTION ----------------------------
\section{Introduction}

Interactive segmentation algorithms allow users to explicitly control the predictions using interactive input at several iterations, in contrast to common semantic and instance segmentation algorithms that can only input an image and output a segmentation mask in one pass. Such interaction makes it possible to select an object of interest and correct prediction errors. Another important feature of this group of algorithms is the capability to segment objects of previously unseen classes.

User input can be formalized via various representations: scribbles, clicks, extreme points, etc. Click-based interactive segmentation is one of the most well-studied topics among other deep learning-based interactive segmentation approaches, as it has well-established protocols of training and evaluation \cite{xu2016deep, jang2019interactive, kontogianni2020continuous, sofiiuk2020f}. It also employs quite an intuitive and simple way to specify the desired object. Scribble-based methods often use heuristic and complicated procedures for simulating user input, which makes it hard to fairly evaluate them \cite{bai2014error, lin2016scribblesup, freedman2005interactive}. Approaches based on extreme points are not intuitive for users and are not flexible enough due to a limited number of user interactions \cite{maninis2018deep}. In our work, we focus on click-based interactive segmentation.

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{figs/spotlight.jpg}
    \caption{Besides segmenting new objects, proposed method allows to correct external masks, \eg produced by other instance or semantic segmentation models. A user can fix false negative and false positive regions with positive (green) and negative (red) clicks, respectively.}
    \label{fig:external_mask_correction}
\end{figure}

In general, the development of deep learning-based algorithms for semantic and instance segmentation requires a huge amount of annotated data.  Annotating data with segmentation masks is very time-consuming and therefore expensive. Interactive segmentation can significantly simplify and speed up this process \cite{acuna2018efficient, benenson2019large, agustsson2019interactive}, which is one of the most important applications of interactive segmentation. It can also be used in photo editing, allowing users to select objects easily, which is especially important for smartphone applications where user input is often limited by finger or stylus activities.

Recent works on click-based interactive segmentation propose complicated inference-time optimization procedures to improve the quality of interactive segmentation even further \cite{jang2019interactive, kontogianni2020continuous, sofiiuk2020f}. The deployment of these models is substantially limited, for instance, on mobile devices, as it requires the implementation of backward passes with gradients that are not provided by popular frameworks. Surprisingly, we find that properly chosen baselines of click-based interactive segmentation models without any without any explicit refinement techniques and other specific modifications can be trained using the standard random points sampling procedure and show state-of-the-art performance. We observe that modifications of poorly tuned baselines can provide false performance improvements, while strong baselines with the same modifications do not show any improvement at all. At the same time, datasets with coarse masks or containing a small amount of images may become a bottleneck for the performance of interactive segmentation models. Our experiments show that usage of diverse large datasets with fine masks for training plays a crucial role for the performance of discussed models. Therefore, we choose a strong baseline model for implementing and evaluating new modifications. We train models on a combination of LVIS and COCO datasets \cite{lin2014microsoft, gupta2019lvis}, that, to the best of our knowledge, are the most suitable for training interactive segmentation models.

We propose an extension of click-based interactive segmentation that allows to modify existing instance segmentation masks interactively. We revive an iterative training procedure, and make a network aware of the mask from a previous step \cite{mahadevan2018iteratively}. We show that such awareness improves the models' stability, i.e. allows to avoid accuracy dropping when adding new clicks. We propose a new training dataset obtained by combining the LVIS and COCO datasets.


%%%%%%%%% ---------------------------- RELATED WORK ----------------------------
\section{Related Work}
\label{sec:related}

\textbf{Interactive segmentation methods.} Interactive segmentation is a longstanding problem in computer vision. Early methods \cite{boykov2001interactive, grady2006random, gulshan2010geodesic, rother2004grabcut} tackle the problem using optimization-based approaches minimizing a specifically constructed cost function defined on a graph over image pixels. GrabCut proposed in \cite{rother2004grabcut} is a classic approach based on iterative energy minimization of a cost function, that is modeled using a Gaussian mixture. Xu~\etal~\cite{xu2016deep} first propose a CNN-based model for interactive segmentation and introduced a clicks simulation strategy for training that is adopted in some further works. Later, \cite{li2018interactive, liew2017regional, liew2019multiseg, lin2020interactive} propose various CNN-based methods for interactive segmentation that aimed at the refinement of predictions by increasing the diversity of predicted masks, and using the attention mechanism. The novel subgroup of methods emerges with an introduction of the Backpropagating Refinement Scheme (BRS) in \cite{jang2019interactive}. The authors propose an optimization procedure that minimizes a discrepancy between the predicted mask and the map of input clicks after each click with respect to the input distance maps. This refinement technique improves the segmentation quality at the cost of increasing the runtime. Sofiiuk~\etal~\cite{sofiiuk2020f} address this issue by proposing f-BRS, a lightweight version of BRS that uses a similar optimization procedure, though with respect to internal parameters introduced to the higher levels of a network, reducing runtime on an order of magnitude. Kontogianni~\etal~\cite{kontogianni2020continuous} introduce another method of test time refinement targeting the optimization process on network parameters.

\textbf{Different types of interactive feedback.} While clicks are the mainstream form of input in interactive segmentation, a lot of works explore other variations of interactive feedback from the user. For example, one of the simplest types of interactions is a bounding box, that is used in \cite{cheng2015densecut, rother2004grabcut, wu2014milcut}. The main drawbacks of these approaches are lack of specific object reference inside the selected area and lack of interface for correction of the predicted mask. The limitations of bounding box-based interaction are addressed in \cite{zhang2020interactive}. The authors propose to combine clicks with bounding boxes to provide more specific object guidance and allow corrections of the predicted mask. DEXTR \cite{maninis2018deep} uses extreme points of the target object, i.e. left-most, right-most, top, and bottom pixels as an input. On the one hand, such an interaction is compact and is limited by 4 clicks. On the other hand, placing extreme points in the right locations is harder than making an arbitrary click on an object and there is no support for corrections, similar to interaction via bounding boxes. Scribbles are used in many early works \cite{bai2014error, boykov2001interactive, grady2006random, gulshan2010geodesic, kim2008generative}. This type of feedback provides richer prior information for the algorithm, compared to the others. In contrast, putting a stroke requires more effort from the user, compared to simpler forms of interactions. Another drawback that refrains CNN-based methods from the wide adoption of scribbles is that realistic strokes simulation for training the neural networks is a rather hard task. Therefore, the vast majority of scribble-based methods employ graphical models or similar training-free techniques. Apart from the aforementioned forms of interaction, some works use a combination of traditional input representation with other modalities, e.g. PhraseClick \cite{ding2020phraseclick} combines clicks with text input to better infer the attributes of the target object, thus requiring fewer clicks.

\textbf{Segmentation mask refinement.} The majority of segmentation mask refinement techniques are targeted to refine the boundaries of a mask by making local corrections without changing the mask globally, i.e. excluding or including large parts of an object. These methods can either be a part of the network architecture \cite{pinheiro2016learning, yuan2020segfix} or serve as a post-processing step \cite{chen2017deeplab, zhou2020deepstrip}. Local and boundary refinement tasks are often addressed without any additional input from the user, as all the required features are automatically extracted from the input image and mask. Global refinement, on the contrary, may require additional feedback from the user to make improvements to the mask. We propose to combine interactive segmentation and mask refinement allowing to refine external segmentation mask with user clicks as guidance.

%%%%%%%%% ------------------------ PROPOSED METHOD ----------------------------
\section{Proposed Method}
\label{sec:proposed}

\subsection{Revising Network Architecture}
\label{sec:revising_network_and_brs}

\begin{figure}
    \centering
    \includegraphics[width=0.6\linewidth]{figs/distmaps.png}
    \caption{Visualization of two different approaches for encoding user clicks.}
    \label{fig:clicks_encodings}
\end{figure}

The task of interactive segmentation is very similar to instance or semantic segmentation in terms of the network architecture. In these tasks, networks take high-resolution input and produce high-resolution segmentation masks aligned with the input. The key difference is in the user input: its main aspects are the encoding and processing of the encoded input inside the network. Therefore, there is no need to reinvent general segmentation architectures. Instead, it is reasonable to rely on time-tested state-of-the-art segmentation networks and focus on interactive-specific parts.

We consider the DeepLabV3+~\cite{chen2018encoder} and HRNet+OCR~\cite{WangSCJDZLMTWLX19, yuan2019object} semantic segmentation architectures as a backbone for our interactive segmentation model. While DeepLabV3+ is a well-studied segmentation architecture that proved efficient in many segmentation-related tasks, HRNet is a relatively new promising one that was specially designed to produce high-resolution output. According to our experiments, HRNet is a more preferable architecture for this task. We provide the ablation study for the backbone architecture in Section~\ref{sec:ablation} and show the results in Tables~\ref{tab:ablation_architecture}, \ref{tab:ablation_datasets}~and~\ref{tab:arch_comparison}. 

\begin{figure}[t]
    \centering
    \includegraphics[width=0.6\linewidth]{figs/input_schemes.png}
    \caption{Different architecture choices of feeding encoded clicks to a backbone, described in Section~\ref{sec:revising_network_and_brs}.}
    \label{fig:input_schemes}
\end{figure}

\textbf{Clicks encoding.} There are positive and negative clicks in click-based interactive segmentation. These clicks are represented by their coordinates in an image. In order to feed them to a convolutional network, we should encode them in a spatial form first. Encoding the clicks via a distance transform from Xu~\etal~\cite{xu2016deep} is the most common approach to clicks encoding. However, they can also be represented by gaussians or disks with a fixed radius. Benenson~\etal~\cite{benenson2019large} perform a detailed ablation study on clicks encoding and find that disks with a small radius surpass the other encodings in terms of model performance.

We conduct our ablation study to compare the distance transform encoding with the disks encoding and find that the latter outperforms the former. We assume that the following observation explains the superiority of disks. The changes in disk encoding caused by adding new points or moving existing ones are always local and only slightly affect the encoding map. At the same time, a distance transform map can change drastically when a new point is added, especially if there are only a few points. In turn, such sudden considerable changes might confuse a network. We show a visualization of both the distance transform and disk encodings in Figure~\ref{fig:clicks_encodings}.

\textbf{Feeding encoded clicks to a backbone.} Semantic segmentation backbones are usually pre-trained on ImageNet~\cite{deng2009imagenet} and  can take only RGB images as an input. The most common way to handle additional input, e.g. encoded user clicks, is to augment the weights of the first convolutional layer of a pre-trained model to accept N-channel input instead of only an RGB image~\cite{xu2016deep, benenson2019large}. In our work, we denote this modification of network architecture by Conv1E. Sofiiuk~\etal~\cite{sofiiuk2020f} propose the Distance Maps Fusion (DMF) module which transforms an image concatenated with additional user input channels into 3-channel input.

We propose a new simple approach to solving this problem, which is described as follows. We introduce a convolutional block that outputs the tensor of exactly the same shape as the first convolutional block in the backbone does. This tensor is then summed element-wise with the output of the first backbone convolutional layer, which usually has 64 channels. We denote this modification of network architecture by Conv1S. While being similar to Conv1E, it allows to choose a different learning rate for new weights without affecting the weights of a pre-trained backbone. We show the schemes of these architectural modifications in Figure \ref{fig:input_schemes} and provide an ablation study in Section \ref{sec:ablation}.

% Table: comparison of architectures
\begin{table}
\footnotesize
\begin{center}
\caption{Ablation studies of the network architecture choices described in Section \ref{sec:revising_network_and_brs}. Each cell consists of two results "X/Y", where "X" and "Y" correspond to evaluation without and with \mbox{f-BRS-B\cite{sofiiuk2020f}}, respectively. "DT" stands for the distance transform clicks encoding. All models are trained on SBD.}
\begin{tabular}{l|c|c|c|c}
\hline
\multirow{2}{*}{Backbone} & 
\multirow{2}{*}{\begin{tabular}{@{}c@{}} Input \\ Scheme \end{tabular}} & 
\multirow{2}{*}{\begin{tabular}{@{}c@{}} Clicks \\ Encoding\end{tabular}} &
\multicolumn{2}{c}{NoC\textsubscript{20}@90} \\
\cline{4-5}
{} & {} & {} & Berkeley & DAVIS \\
\hline
\hline
\multirow{5}{*}{ResNet-34}        & DMF \cite{sofiiuk2020f}    & DT      & 5.50/4.32 & 8.45/8.34 \\
                                  & Conv1E & DT                & 4.79/4.43 & 7.56/7.60 \\
                                  & Conv1S & DT                & 4.98/4.16 & 7.41/7.28 \\
                                  & Conv1S & Disk3             & 4.52/4.04 & 7.27/7.18 \\
                                  & Conv1S & Disk5             & 4.09/3.89 & 6.92/7.22 \\
\hline
\multirow{5}{*}{HRNet-18}    & DMF \cite{sofiiuk2020f}   & DT      & 4.93/4.35 & 8.59/8.00 \\
                                 & Conv1E                & DT      & 4.41/3.95 & 7.50/7.43 \\
                                 & Conv1S                & DT      & 3.99/3.81 & 7.16/7.24 \\
                                 & Conv1S                & Disk3   & 3.63/3.47 & 7.14/7.04 \\ 
                                 & Conv1S                & Disk5   & 3.52/3.50 & 6.90/6.97 \\
\hline

\end{tabular}
\label{tab:ablation_architecture}
\end{center}
\end{table}

\subsection{Iterative Sampling Strategy}
\label{sec:iterative_sampling}

Most recent works on click-based interactive segmentation use the sampling strategy for simulating user clicks during training, where a set of positive and negative clicks is randomly generated without considering any relations between them \cite{sofiiuk2020f, lin2020interactive, benenson2019large, xu2016deep}. In practice, every new click is placed in the erroneous region of a prediction produced by a network using the set of previous clicks. This fact is completely ignored in the random sampling strategy. It also makes it impossible to integrate masks from previous interactions into a model, as we need to have ordered interactions and a sequence of corresponding predictions to successfully train such a model.

The iterative sampling strategy, which resembles the interaction with a real user, is employed to a certain extent in several works \cite{mahadevan2018iteratively, majumder2019content, kontogianni2020continuous}. As full iterative sampling is very computationally expensive, in these works random sampling is used for initialization and then a few clicks are added using the iterative sampling procedure.

We adopt the iterative sampling procedure proposed by Mahadevan~\etal~\cite{mahadevan2018iteratively} with the following changes. First, we sample each point not just from the center of a mislabelled region, but from the region obtained by applying morphological erosion operation to the mislabelled region, so that the eroded region has 4 times less area. We observe that choosing center points leads to overfitting to the NoC evaluation metric (see Section \ref{sec:experiments} for the details). In practice, however, these models demonstrate worse performance and unstable behavior when a user places a click near the borders of an object or a mislabelled region. Second, we do not save simulated clicks for dataset samples during training and simulate user clicks for each batch individually. For that reason, we limit the maximum number of sampling iterations to $N_{iters}$, and each batch can uniformly get from~0~to~$N_{iters}$ iterations.

We use a combination of the random and iterative sampling strategies for training our iterative models. First, we simulate user clicks with the random sampling strategy \cite{xu2016deep} just as we do for our non-iterative baselines. Then, we add from~0~to~$N_{iters}$ iteratively simulated clicks. We provide an ablation study of the number of iteratively sampled clicks in Section \ref{sec:ablation}.

\subsection{Incorporating Masks From Previous Steps}
\label{sec:masks_from_previous_steps}

In interactive segmentation, it seems natural to incorporate output segmentation masks from previous interactions as an input for the next correction, providing additional prior information that can help improve the quality of prediction. In case of incorporating the mask from a previous interaction, it is necessary to use the iterative sampling for simulating user interactions during training. Along with the iterative sampling, Mahadevan~\etal~\cite{mahadevan2018iteratively} propose passing an output mask from a previous iteration into the model as an optional extension of their method.

To train a model with the mask from a previous step, we use a combination of the random and iterative user interactions simulation described in Section~\ref{sec:iterative_sampling}. Our model takes this mask as the third channel together with two channels for positive and negative encoded clicks, respectively. For the first interaction, as well as for the batches with skipped iterative sampling, we feed an empty mask to our model.

\subsection{Normalized Focal Loss}
\label{sec:normalized_focal_loss}
Binary cross entropy (BCE) loss is one of the standard loss functions for training semantic segmentation algorithms. Besides, some state-of-the-art interactive segmentation methods \cite{jang2019interactive, liew2019multiseg, majumder2019content, maninis2018deep, zhang2020interactive} adopt it for training the network. The main drawback of BCE is that it treats all examples equally, slowing the training during later epochs as the gradient from almost correctly segmented areas propagates similarly to the gradient from erroneous regions. Focal loss was introduced in \cite{lin2017focal} to alleviate this problem. Let $\hat{M}$ denote the output of the network and $p_{i,j}$ denote the confidence of prediction at the point $(i,j)$. Then the FL is formulated as follows:


\begingroup

%\setlength\abovedisplayshortskip{2pt}
%\setlength\belowdisplayshortskip{6pt}
\setlength\abovedisplayskip{0pt}
\setlength\belowdisplayskip{6pt}
\begin{equation}
    FL(i,j)=-(1-p_{i,j})^{\gamma}\log p_{i,j}
\label{eq:FL}
\end{equation}

\endgroup

One can notice that the total weight $P(\hat{M})=\sum_{i,j}(1-p_{i,j})^\gamma$ decreases when the accuracy of the prediction increases. This means that the total gradient of the FL fades over time, slowing the training process. To mitigate this problem, \cite{sofiiuk2019adaptis} proposed normalized focal loss (NFL) that is formulated as follows:

\begingroup
\setlength\abovedisplayskip{0pt}
\setlength\belowdisplayskip{6pt}
\begin{equation}
    NFL(i,j,\hat{M})=-\frac{1}{P(\hat{M})}(1-p_{i,j})^{\gamma}\log p_{i,j}
\label{eq:NFL}
\end{equation}
\endgroup

The gradient of NFL does not fade over time due to normalization and remains equal to the total gradient of BCE. This allows for faster convergence and better accuracy compared to training with BCE. We choose NFL as a loss function for training the proposed methods and provide an ablation study with respect to loss function choice in Section~\ref{sec:ablation} (results are provided in Table~\ref{tab:ablation_loss}).

% Table: comparison of loss functions
\begin{table}
\small
\begin{center}
\caption{Evaluation of the non-iterative baseline method with the \mbox{HRNet-18+OCR} backbone trained on COCO+LVIS (proposed in Section \ref{sec:coco_lvis}) with different loss functions. We report NoC@90 on four datasets.}
\begin{tabular}{l|c|c|c|c}
\hline

\multirow{2}{*}{Method} &
\multicolumn{4}{c}{NoC\textsubscript{20}@90} \\
\cline{2-5}
{} & GrabCut & Berkeley & SBD & DAVIS \\
\hline
\hline
BCE      & 1.82 & 3.13 & 7.58 & 6.31 \\
Soft IoU & 2.02 & 3.03 & 7.94 & 6.45 \\
FL       & 1.80 & 3.28 & 7.56 & 6.40 \\
NFL      & \textbf{1.70} & \textbf{2.48} & \textbf{6.72} & \textbf{5.90} \\
\hline

\end{tabular}
\label{tab:ablation_loss}
\end{center}
\end{table}



%%%%%%%%% ------------------------ COCO+LVIS for IS ---------------------------
\section{Dataset for Interactive Segmentation}
\label{sec:dataset_for_is}

\subsection{Reviewing Existing Datasets}
\label{sec:existing_datasets}
The vast majority of recent CNN-based interactive segmentation methods \cite{jang2019interactive, li2018interactive, liew2017regional, liew2019multiseg, majumder2019content, sofiiuk2020f, xu2016deep} are trained either using Semantic Boundaries Dataset \cite{hariharan2011semantic}, or the Pascal~VOC dataset \cite{everingham2010pascal}, or the combination of these two datasets, as they share the same set of images. The augmented dataset contains a total of 10582 images with 25832 instance-level masks. The total number of classes in this dataset is twenty: 7 categories of transportation, 6 species of animals, 6 types of indoor objects, and a separate class for persons. These classes cover only general types of objects, implying limitations on the variety of predictable classes.
Recently, large-scale segmentation datasets OpenImages \cite{benenson2019large, kuznetsova2018open} and LVIS \cite{gupta2019lvis} were introduced. These datasets stimulated the development of state-of-the-art instance segmentation algorithms \cite{hu2020learning, jiawei2020balanced, tan2020equalization, wang2020solov2}, providing a large variety of labeled classes with a sufficient number of examples. LVIS contains around 1.2M instances on 100k images embracing more than a thousand object classes, while OpenImages has 2.6M instances on 944k images covering 350 object categories. We believe that having such a diverse dataset is one of the key components in training of the state-of-the-art interactive segmentation model.
Another important characteristic of a segmentation dataset is annotation quality. Gupta~\etal~\cite{gupta2019lvis} present a study of annotation quality in some of the instance segmentation benchmarks, comparing a subset of masks from each dataset to experts’ annotations. According to this study, the LVIS dataset has the highest annotation quality among reported datasets, presumably allowing to achieve higher prediction quality.


\subsection{Combination of COCO and LVIS}
\label{sec:coco_lvis}
Considering the observations from Section~\ref{sec:existing_datasets}, the LVIS dataset appears to be the best choice for training the models except for one drawback: it is long-tailed and therefore lacking general object categories, which can affect the accuracy and generalization of the trained model. As a solution to this problem, we propose to augment LVIS labels with masks from the COCO \cite{lin2014microsoft} dataset, as these two datasets share the same set of images. The COCO segmentation dataset contains a total of 1.2M instance masks on 118k training images with 80 object classes. These categories represent more common and general objects, complementing the long-tailed object classes from the LVIS dataset. We propose the following procedure to construct the combined COCO+LVIS dataset with diverse object classes from the most common to less frequent ones. All masks from both datasets are joined together except for those masks from COCO that have a corresponding mask from LVIS with an intersection over union (IoU) score between them larger than 80\%. In that case, we only keep the mask from LVIS, since it presumably has better overall and especially boundary quality.
As a result of the described procedure, we obtained a dataset with 104k images and 1.6M instance-level masks. We argue that the further development of interactive segmentation algorithms relies heavily on the training data. Thus, we provide the comparative study of the training datasets in Section \ref{sec:ablation}, results are provided in Table~\ref{tab:ablation_datasets}. Based on our experiments and the aforementioned observations, we conclude that the proposed COCO+LVIS dataset is the best choice for training interactive segmentation methods.

% Table: comparison of training datasets
\begin{table}
\footnotesize
\begin{center}
\caption{Evaluation results for the non-iterative baseline method with the \mbox{ResNet-34} and \mbox{HRNet-18+OCR} backbones trained on various datasets. NoC@90 is reported.}
\begin{tabular}{l|l|c|c|c|c}
\hline
\multirow{2}{*}{Train Dataset} &
\multirow{2}{*}{Backbone} & 
\multicolumn{4}{c}{NoC\textsubscript{20}@90} \\
\cline{3-6}
{} & {} & GrabCut & Berkeley & SBD & DAVIS \\
\hline
\hline
\multirow{2}{*}{ADE20k}        & ResNet-34    & 2.70 & 5.09 & 8.27 & 8.56 \\
                               & HRNet-18     & 2.68 & 4.78 & 8.02 & 8.32 \\
\hline
\multirow{2}{*}{OpenImages}    & ResNet-34    & 2.20 & 4.68 & 8.16 & 7.47 \\
                               & HRNet-18     & 2.02 & 4.47 & 7.95 & 8.08 \\
\hline
\multirow{2}{*}{SBD}           & ResNet-34    & 2.94 & 4.73 & 6.94 & 7.56 \\
                               & HRNet-18     & 2.41 & 3.95 & 6.66 & 7.17 \\
\hline
\multirow{2}{*}{Pascal~VOC+SBD} & ResNet-34   & 2.77 & 4.53 & 6.92 & 6.48 \\
                                & HRNet-18    & 2.25 & 3.63 & 6.63 & 6.16 \\
\hline
\multirow{2}{*}{LVIS}          & ResNet-34    & 2.59 & 3.61 & 7.99 & 6.98 \\
                               & HRNet-18     & 2.44 & 3.13 & 8.14 & 7.18 \\
\hline
\multirow{2}{*}{COCO}          & ResNet-34    & 1.80 & 3.34 & 6.29 & 6.11 \\
                               & HRNet-18     & 1.77 & 2.90 & 6.32 & 5.85 \\
\hline
\multirow{2}{*}{COCO+LVIS}     & ResNet-34    & 1.74 & 2.91 & 6.53 & 6.01 \\
                               & HRNet-18     & 1.70 & 2.48 & 6.86 & 6.00 \\
\hline

\end{tabular}
\label{tab:ablation_datasets}
\end{center}
\end{table}

%%%%%%%%% ---------------------------- EXPERIMENTS ----------------------------
\section{Experiments}
\label{sec:experiments}

We perform an extensive evaluation of the proposed approach by conducting ablation studies for all sufficient parts of the method, exploring the convergence properties with an increasing number of clicks and comparing our method with current state-of-the-art works.

\textbf{Datasets.}
We evaluate the performance of our method on five common benchmarks for interactive segmentation with instance-level annotations. The GrabCut \cite{rother2004grabcut} dataset contains 50 images with a single object mask for each image. We adopt the test subset of Berkeley \cite{martin2001database} introduced in \cite{mcguinness2010comparative}, which consists of 100 masks for 96 images. The DAVIS \cite{perazzi2016benchmark} dataset was introduced for the evaluation of video segmentation datasets. We use the subset of 345 randomly sampled frames of video sequences that was introduced in \cite{jang2019interactive} for evaluation. We follow the common protocol and combine all instance-level masks for one image into one segmentation mask. We also use the validation part of the Pascal VOC \cite{everingham2010pascal} dataset that consists of 1449 images with 3417 instances. Each instance mask is used separately during evaluation. The Semantic Boundaries Dataset (SBD) \cite{hariharan2011semantic} contains 6671 instance-level masks for 2820 images. This dataset has been used for evaluating the interactive segmentation algorithms since \cite{xu2016deep}. To test the algorithm, we use each of instance masks separately and do not combine them into one segmentation mask for one image.

\textbf{Evaluation metric.}
We perform the evaluation using the standard Number of Clicks (NoC) measure, reporting the number of clicks required to achieve the predefined Intersection over Union (IoU) threshold between predicted and ground truth masks. We denote NoC with IoU threshold set to 85\% and 90\% as NoC@85 and NoC@90, respectively. To generate clicks during the evaluation procedure, we follow the strategy used in \cite{li2018interactive, xu2016deep}. The next click is placed at the center of the region with any type of prediction error (false positive or false negative) with the largest area among other erroneous regions. The region center is defined as the point farthest from the boundaries of the corresponding region.

% Table: comparison of models capacity and complexity
\begin{table}
\small
\begin{center}
\caption{Comparison of different models in terms of the number of FLOPs and parameters. All models take an image with a~resolution of $400\times400$ to compute the number of FLOPs. We use exactly the same architecture of the DeepLab models as it was proposed in \cite{sofiiuk2020f}.}
\begin{tabular}{l|cc|cc}
\hline

Model & \#Params & Ratio-to-HRNet18 & \#FLOPs & Ratio-to-HRNet18 \\
\hline
\hline
HRNet18s+OCR           & 4.22M  & 0.4x & 17.84G  & 0.6x   \\
HRNet18+OCR            & 10.03M & 1.0x & 30.80G   & 1.0x   \\
HRNet32+OCR            & 30.95M & 3.1x & 82.84G  & 2.7x   \\
DeepLab-ResNet-34      & 19.17M & 1.9x & 122.28G & 4.0x   \\
DeepLab-ResNet-50      & 31.40M & 3.1x & 170.13G & 5.5x   \\
\hline

\end{tabular}
\label{tab:arch_comparison}
\end{center}
\end{table}

\textbf{Segmentation backbones.}
We consider two different segmentation architectures: DeepLab-V3+ \cite{chen2018encoder} with ResNet and HRNet+OCR \cite{WangSCJDZLMTWLX19, yuan2019object}. To implement DeepLab-V3+ for interactive segmentation we follow \cite{sofiiuk2020f}. We use the implementation of HRNet and models pre-trained on ImageNet presented in the official repository\footnote{\url{https://github.com/HRNet/HRNet-Image-Classification}}. There are several versions of HRNet which differ in terms of their capacity: HRNet-W18-C, HRNet-W30-C, HRNet-W32-C, etc. In our work we employ HRNet-W18-C-Small-v2, HRNet-W18-C, HRNet-W32-C, referring to them as HRNet-18s, HRNet-18, HRNet-32 for the sake of brevity. The numbers of parameters and FLOPs of our models is shown in Table \ref{tab:arch_comparison}. All the networks are initialized by the weights pre-trained on ImageNet weights.

We use the OCR module proposed in \cite{yuan2019object} for all HRNets. The original OCR module always produces 512 output channels regardless of the size of a backbone. We proportionally reduce the width of all the OCR layers, so that it produces 48, 64 and 128 channels for HRNet-18s, HRNet-18 and HRNet-32, respectively.

\textbf{Implementation details.}
The training task is binary segmentation with normalized focal loss, described in Section \ref{sec:normalized_focal_loss}, as an objective. We use image crops with a size of $320\times480$ and randomly resize images with a scale factor from 0.75 to 1.40 before cropping. We use horizontal flip and random jittering of brightness, contrast, and RGB values as augmentations during training. We adopt test time augmentations from f-BRS method \cite{sofiiuk2020f}, using Zoom-In technique and averaging predictions from original and horizontally flipped image during evaluation.

In all our experiments, we use Adam optimizer with $\beta_1 = 0.9, \beta_2 = 0.999$ and train the models for 55 epochs on the proposed COCO+LVIS dataset (49 epochs with learning rate $5 \times 10^{-4}$, then learning rate is decreased by 10 times on 50th and 53rd epochs). The learning rate for backbone networks is set 10 times lower than the learning rate for the rest of the network. We denote one pass through each image of the dataset as an epoch. The batch size is set to 32. We train the networks based on HRNet-18s, HRNet-18 and HRNet-32 on 1, 2 and 4 Tesla P40, respectively. To train the model based on ResNet-34, we use 1 Tesla P40.

We implement the proposed method using the PyTorch \cite{paszke2019pytorch} framework.

\subsection{Convergence Analysis}
\label{sec:convergence}

One of the key properties of an interactive segmentation algorithm is convergence to sufficient accuracy with an increasing number of clicks.  Previous works \cite{jang2019interactive, kontogianni2020continuous, sofiiuk2020f} improve the convergence using inference time optimization schemes, that force the predictions to match with input clicks. Our method is free from any refinement schemes and makes use of the proposed architecture and the iterative training scheme to achieve the desired level of convergence.

Sofiiuk~\etal~\cite{sofiiuk2020f} introduced the evaluation protocol for analysis of convergence for interactive segmentation methods. The main aspect of this protocol is the NoC\textsubscript{100} evaluation metric, which is similar to the NoC described earlier but has the clicks limit set to 100. Such size of interactive feedback should give an algorithm enough information to converge. In case the algorithm can not reach the IoU threshold, the image is supposedly too hard for the method to handle, therefore increasing the click limit is unlikely to change it.

Table~\ref{tab:convergence} shows the comparison of the the results of BRS \cite{jang2019interactive}, f-BRS-B \cite{sofiiuk2020f} and the proposed method with HRNet18+OCR backbone, reporting NoC\textsubscript{100}, number of images for which the 90\% threshold on IoU was not achieved after 20 and 100 clicks. All models are trained on the SBD \cite{hariharan2011semantic} dataset for a fair comparison.

% Table: convergence analysis
\begin{table}
\footnotesize
\begin{center}
\caption{Convergence analysis on Berkeley, SBD and DAVIS. We report the number of images that were not correctly segmented after 20 and 100 clicks and the NoC\textsubscript{100}@90 metric. All models are trained on SBD.}
\begin{tabular}{l|l|c|c|c}
\hline 
Dataset & Model & 
\begin{tabular}{@{}c@{}}\#images \\ $\geq$20 \end{tabular} & 
\begin{tabular}{@{}c@{}}\#images \\ $\geq$100 \end{tabular} & 
\begin{tabular}{@{}c@{}} NoC\textsubscript{100} \\  @90 \end{tabular} \\
\hline 
\hline
\multirow{4}{*}{Berkeley} & BRS \cite{jang2019interactive}  & 10 & 2 & 8.77 \\
                          & f-BRS-B \cite{sofiiuk2020f}     &  \textbf{2} & \textbf{0} & \textbf{4.47} \\
                          & HRNet-18                        &  7 & 3 & 7.10 \\
                          & HRNet-18 ITER-M                 &  3 & 2 & 4.89 \\
\hline
\multirow{4}{*}{DAVIS}    & BRS \cite{jang2019interactive}  & 77 & 51 & 20.98 \\
                          & f-BRS-B \cite{sofiiuk2020f}     & 78 & 50 & 20.70  \\
                          & HRNet-18                        & 67 & 50 & 19.83 \\
                          & HRNet-18 ITER-M                 & \textbf{57} & \textbf{44} & \textbf{18.42} \\
\hline
\multirow{3}{*}{SBD}      & f-BRS-B \cite{sofiiuk2020f}     & 1466 & 265 & 14.98 \\
                          & HRNet-18                        & 1051 & 450 & 13.62 \\
                          & HRNet-18 ITER-M                 &  \textbf{671} & \textbf{215} & \textbf{9.52} \\
\hline

\end{tabular}
\label{tab:convergence}
\end{center}
\end{table}

\subsection{Ablation Studies}
\label{sec:ablation}

% Table: iterative ablation
\begin{table}
\footnotesize
\begin{center}
\caption{Ablation studies on the maximum number of iterations $N_{iters}$ in the iterative sampling. All the results are reported for the model with the HRNet-18+OCR backbone trained on COCO+LVIS.}
\begin{tabular}{c|c|c|c|c}
\hline 
\multirow{2}{*}{$N_{iters}$} & Prev &  \multicolumn{3}{c}{NoC\textsubscript{20}@90} \\
\cline{3-5}
{} & {Mask} & Berkeley & DAVIS & SBD \\
\hline 
\hline

3 & -- & 2.38 & 5.92 & 6.49 \\
3 & + & 2.26 & 5.74 & 6.06 \\
\hline
1 & + & 2.57 & 5.81 & 6.15 \\
2 & + & 2.48 & \textbf{5.70} & 6.10 \\
3 & + & \textbf{2.26} & 5.74 & 6.06 \\
4 & + & 2.52 & 6.03 & \textbf{6.04} \\
5 & + & 2.49 & 5.98 & 6.24 \\
6 & + & 2.55 & 6.11 & 6.82 \\
\hline

\end{tabular}
\label{tab:iterative}
\end{center}
\end{table}

\textbf{Network architecture ablations.}
In Section \ref{sec:revising_network_and_brs}, we discuss different architectural choices for interactive segmentation networks. First, we explore different strategies of feeding encoded clicks to the model (DMF, Conv1E and Conv1S). We find that the HRNet-18 and ResNet-34 models with Conv1S show better performance in general and we use it in all further experiments. Then, we evaluate different clicks encoding strategies. We find that changing the distance transform encoding to the disk encoding significantly improves results of both HRNet-18 and ResNet-34, which confirms the findings of Benenson~\etal~\cite{benenson2019large} whose model performed best with disks with a radius of 3. However, our experiments show that a radius of 5 is better than 3. The results of all ablations can be found in Table~\ref{tab:ablation_architecture}.

In addition, these experiments clearly demonstrate the superiority of HRNet-18 over DeepLabV3+ with ResNet-34, especially when the former has several times fewer parameters and FLOPs than the latter (the number of parameters and FLOPs can be found in Table~\ref{tab:arch_comparison}). We use HRNet with the Conv1S input scheme and encode clicks with disks of radius 5 in all other experiments.

% Figure: a comparison between non-iterative and iterative models with and w/o prev mask
\begin{figure}[h]

\centering
\includegraphics[width=.48\textwidth]{figs/ious_ablation_mask/GrabCut.png}
\includegraphics[width=.48\textwidth]{figs/ious_ablation_mask/Berkeley.png}
\includegraphics[width=.48\textwidth]{figs/ious_ablation_mask/DAVIS.png}
\includegraphics[width=.48\textwidth]{figs/ious_ablation_mask/SBD.png}

\caption{Mean IoU@$k$ for varying number of clicks $k$ on GrabCut, Berkeley, DAVIS and SBD. The iterative model that takes a mask from a previous step is much more stable and converges to a better~IoU. All the results are reported for the model with the HRNet-18+OCR backbone trained on COCO+LVIS, iterative models are trained with $N_{iters}=3$. }
\label{fig:ious_ablation_mask}
\end{figure}

\textbf{Training datasets comparison.} To further research the findings from Section~\ref{sec:dataset_for_is}, we evaluate the models trained on each of the six common segmentation datasets: ADE20k \cite{zhou2017scene}, OpenImages \cite{benenson2019large, kuznetsova2018open}, SBD \cite{hariharan2011semantic}, Pascal~VOC \cite{everingham2010pascal} augmented with labels from SBD (denoted as Pascal~VOC+SBD), LVIS \cite{gupta2019lvis}, COCO \cite{lin2014microsoft} and also the COCO+LVIS dataset, proposed in Section~\ref{sec:coco_lvis}. We provide results for non-iterative baseline methods with HRNet-18+OCR and ResNet-34 backbones, trained with NFL loss. We report NoC@90 on Berkeley, SBD and DAVIS in Table~\ref{tab:ablation_datasets}. The performance on COCO and COCO+LVIS is relatively close, but the model trained on the proposed dataset generalizes better due to wider class distribution in the training set. Another observation is that the models trained on SBD and Pascal~VOC+SBD show the best performance on the SBD dataset in terms of NoC@90, which is most likely caused by the similar distribution of training and testing sets. Nonetheless, these models are inferior to the model trained on the COCO+LVIS dataset in terms of performance on other benchmarks.
All models trained for dataset comparison share the same training parameters and augmentations, that are described at the beginning of Section~\ref{sec:experiments}. We train the models for the following number of epochs on each dataset: 180 for ADE20k, 5 for OpenImages, 120 for SBD and Pascal~VOC+SBD, 55 for LVIS and COCO+LVIS, 40 for COCO. We denote one pass through each image of the dataset as an epoch.

\textbf{Loss functions comparison.} We compare four loss functions that were used for training segmentation methods in recent works: binary cross entropy (BCE) loss, focal loss (FL) \cite{lin2017focal}, soft IoU loss \cite{rahman2016optimizing} and normalized focal loss (NFL) \cite{sofiiuk2019adaptis}. We evaluate the performance of the baseline HRNet-18+OCR model trained on the COCO+LVIS dataset, described in Section~\ref{sec:coco_lvis}. The results are presented in Table~\ref{tab:ablation_loss}. We provide only NoC@90 on all standard datasets for simplicity. The evaluation results support the reasoning in Section~\ref{sec:normalized_focal_loss}, demonstrating that training with NFL leads to better accuracy and convergence on all 4 datasets.

\textbf{Iterative training ablations.}
The main hyperparameter of our iterative sampling scheme is the maximum number of iteratively sampled clicks $N_{iters}$ in addition to some set of randomly sampled clicks. Our experiments show that $N_{iters}=3$ is an optimal value of that parameter. Surprisingly, too high values (> 4) lead to instability during training and to worse results. We had to train models with $N_{iters}=5,6$ for several times, as they collapsed after 10-20 epochs of training on COCO+LVIS and showed poor results. We provide the ablation study in Table~\ref{tab:iterative}. 

We study the impact of feeding a mask from the previous click to the iterative model with $N_{iters}=3$. Apart from metrics improvement, we observe substantial improvement of the stability of the model when new clicks are added. As shown in Figure~\ref{fig:ious_ablation_mask}, both the iterative model without a previous mask and the non-iterative model sometimes have drops of mean IoU when the number of clicks increases. It indicates that adding a new click during the process of interactive segmentation can even make the mask worse, contradicting user expectations. This effect is better illustrated on mean IoU plots for small datasets such as GrabCut or Berkeley, as the smoothness of the curve is proportional to the dataset size. When the model relies on a mask from a previous iteration, it can take into account the segmentation result of a previous iteration and avoid unexpected collapse of the current mask. Moreover, it allows to modify existing masks without additional effort. We discover that the trained model can be successfully initialized with an external inaccurate mask without the history of previous clicks to correct the errors of this mask.  Several examples of applying our to this use case are shown in Figure~\ref{fig:external_mask_correction}.

% Figure: Visualisation on Berkeley
\newpage
\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.98\linewidth]{figs/is_vis_h18itm_berkeley.jpg}
    \caption{Visualization of interactive segmentation for the Berkeley images with a different number of clicks fed to the HRNet-18 ITER-M model and obtained by the NoC evaluation procedure \cite{xu2016deep}. Green and red dots denote positive and negative clicks, respectively. There are only 2 images from Berkeley on which our model does not converge to 90\% IoU in 20 clicks. One of them is shown in the third row.}
    \label{fig:is_vis_berkeley}
\end{figure}

\subsection{Comparison with Previous Works}
The quantitative results are summarized in Table~\ref{tab:evaluation_all}. We notice that even the proposed baseline model with HRNet18+OCR backbone outperforms all previous methods. The proposed iterative method that uses a mask from the previous click sets a new state-of-the-art in interactive segmentation on all five benchmarks. Another notable fact is that the smallest among the proposed models with HRNet18s+OCR backbone performs on par with heavier ones, making it possible to use the proposed method on devices with low computational capability.

We provide plots of mean IoU with respect to the number of clicks for the GrabCut, Berkeley, DAVIS and SBD datasets in Figure~\ref{fig:miou_plots}. The proposed method outperforms previous state-of-the-art works and shows great improvement in accuracy and stability while avoiding accuracy drops after any click and converging to a better result.

We also visualise evaluation process of the proposed iteratively trained HRNet-18 model on some images from the Berkeley dataset in Figure~\ref{fig:is_vis_berkeley}. 

% Table: comparison with previous works
\begin{table}
\small
\begin{center}
\caption{Evaluation results on GrabCut, Berkeley, SBD, DAVIS and Pascal VOC datasets. The best and the second best results are set in bold and underlined, respectively. "H18s", "H18" and "H32" stand for "HRNet-18s", "HRNet-18" and "HRNet-32", respectively. Our models with the "IT-M" suffix are iterative models that take a mask from a previous step with $N_{iters}=3$, otherwise they are our non-iterative baselines. The name of the training dataset is indicated below the word "Ours". "C+L" stands for COCO+LVIS.}
\begin{tabular}{ll|c|c|c|c|c|c|c|c}
\hline
\multicolumn{2}{l|}{\multirow{2}{*}{Method}} & \multicolumn{2}{c|}{GrabCut} & Berkeley & \multicolumn{2}{c|}{SBD} & \multicolumn{2}{c|}{DAVIS} & {Pascal VOC} \\
\cline{3-10}
\multicolumn{2}{l|}{} & NoC@85 & NoC@90 & NoC@90 & NoC@85 & NoC@90 & NoC@85 & NoC@90 & NoC@85 \\
\hline
\hline
\multicolumn{2}{l|}{GC \cite{boykov2001interactive}}             &  7.98 & 10.00 & 14.22 & 13.60 & 15.96 & 15.13 & 17.41 & -- \\
\multicolumn{2}{l|}{GM \cite{gulshan2010geodesic}}               & 13.32 & 14.57 & 15.96 & 15.36 & 17.60 & 18.59 & 19.50 & -- \\
\multicolumn{2}{l|}{RW \cite{grady2006random}}                   & 11.36 & 13.77 & 14.02 & 12.22 & 15.04 & 16.71 & 18.31 & -- \\
\multicolumn{2}{l|}{ESC \cite{gulshan2010geodesic}}              &  7.24 &  9.20 & 12.11 & 12.21 & 14.86 & 15.41 & 17.70 & -- \\
\multicolumn{2}{l|}{GSC \cite{gulshan2010geodesic}}              &  7.10 &  9.12 & 12.57 & 12.69 & 15.31 & 15.35 & 17.52 & -- \\
\hline
\multicolumn{2}{l|}{DIOS with GC \cite{xu2016deep}}              &   --  &  6.04 &  8.65 &   --  &   --  &   --  &   --  & 6.88 \\
\multicolumn{2}{l|}{Latent diversity \cite{li2018interactive}}   &  3.20 &  4.79 &   --  &  7.41 & 10.78 &  5.05 &  9.57 & -- \\
\multicolumn{2}{l|}{RIS-Net \cite{liew2017regional}}             &   --  &  5.00 &  6.03 &   --  &   --  &   --  &   --  & 5.12 \\
\multicolumn{2}{l|}{ITIS \cite{mahadevan2018iteratively}}        &   --  &  5.60 &   --  &   --  &   --  &   --  &   --  & 3.80 \\
\multicolumn{2}{l|}{CAG \cite{majumder2019content}}              &   --  &  3.58 &  5.60 &   --  &   --  &   --  &   --  & 3.62 \\
\multicolumn{2}{l|}{BRS \cite{jang2019interactive}}              &  2.60 &  3.60 &  5.08 &  6.59 &  9.78 &  5.58 &  8.24 & -- \\
\multicolumn{2}{l|}{FCA-Net (SIS) \cite{lin2020interactive}}     &   --  &  2.08 &  3.92 &   --  &   --  &   --  &  7.57 & 2.69 \\
\multicolumn{2}{l|}{IA+SA \cite{kontogianni2020continuous}}      &   --  &  3.07 &  4.94 &   --  &   --  &  5.16 &   --  & 3.18 \\
\multicolumn{2}{l|}{f-BRS-B \cite{sofiiuk2020f}}                 &  2.50 &  2.98 &  4.34 &  5.06 &  8.08 &  5.39 &  7.81 & -- \\
\hline
\hline
\multirow{2}{*}{\shortstack{Ours\\ \footnotesize{SBD}}} 
                      & H18                                      & 1.96 & 2.41 & 3.95 & 4.12 & 6.66 & 5.08 & 7.17 & 2.94 \\
                      & H18 IT-M                                 & 1.76 & 2.04 & 3.22 & \textbf{3.39} & \textbf{5.43} & 4.94 & 6.71 & \underline{2.51} \\
\hline
\multirow{4}{*}{\shortstack{Ours\\ \footnotesize{C+L}}}
                      & H18                                      & 1.54 & 1.70 & 2.48 & 4.26 & 6.86 & 4.79 & 6.00 & 2.59 \\
                      & H18s IT-M                                & 1.54 & 1.68 & 2.60 & 4.04 & 6.48 & 4.70 & 5.98 & 2.57 \\
                      & H18 IT-M                                 & \textbf{1.42} & \textbf{1.54} & \underline{2.26} & 3.80 & 6.06 & \underline{4.36} & \underline{5.74} & \textbf{2.28} \\
                      & H32 IT-M                                 & \underline{1.46} & \underline{1.56} & \textbf{2.10} & \underline{3.59} & \underline{5.71} & \textbf{4.11} & \textbf{5.34} & 2.57 \\
\hline

\end{tabular}
\label{tab:evaluation_all}
\end{center}
\end{table}

% Figure: mIoU plots
\begin{figure}[ht]
    \centering
    \includegraphics[width=.48\textwidth]{figs/miou_plots/GrabCut.png}
    \includegraphics[width=.48\textwidth]{figs/miou_plots/Berkeley.png}
    \includegraphics[width=.48\textwidth]{figs/miou_plots/DAVIS.png}
    \includegraphics[width=.48\textwidth]{figs/miou_plots/SBD.png}
    
    \caption{Mean IoU@$k$ for varying number of clicks $k$ on GrabCut, Berkeley, DAVIS and SBD. The iterative models (denoted as ITER-M) show stable performance without accuracy drops and converge to a better IoU. Names of the training datasets are enclosed in parentheses. "C+L" stands for COCO+LVIS.}
    \label{fig:miou_plots}
\end{figure}

\section{Conclusion}

Most of recent studies on interactive segmentation propose models that heavily rely on additional inference-time optimization schemes. In our work, we have demonstrated that a pure feedforward model with a modern backbone architecture can achieve or even surpass current state-of-the-art results. We have introduced a new model able to modify existing segmentation masks as well as segment new objects without any prior masks. It sets a new state-of-the-art on all common interactive segmentation benchmarks.

Our experiments have proved that a training dataset has a major impact on the model performance. We have proposed to combine the two existing instance segmentation datasets, COCO~\cite{lin2014microsoft} and LVIS~\cite{gupta2019lvis}, and use it for interactive segmentation. The resulting large and diverse dataset with high-quality annotations has significantly improved the generalization ability of our model. Training on this dataset allows to push the state-of-the-art results achieved by our model even further.

\noindent\textbf{Acknowledgment.} We thank Julia Churkina for her assistance with editing and for comments that greatly improved the manuscript.

%% Loading bibliography style file
\bibliographystyle{model1-num-names}

% Loading bibliography database
\bibliography{cas-refs}

%\vskip3pt

% \bio{}
% Author biography without author photo.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% Author biography. Author biography. Author biography.
% \endbio

\end{document}

cas-refs.bib
@String(PAMI  = {IEEE TPAMI})
@String(IJCV  = {IJCV})
@String(CVPR  = {CVPR})
@String(ICCV  = {ICCV})
@String(ECCV  = {ECCV})
@String(NIPS  = {NeurIPS})
@String(ICPR  = {ICPR})
@String(BMVC  =	{BMVC})
@String(TOG   = {ACM TOG})
@String(TIP   = {IEEE TIP})
@String(TVCG  = {IEEE TVCG})
@String(TCSVT = {IEEE TCSVT})
@String(TMM   =	{IEEE TMM})
@String(ACMMM = {ACM MM})
@String(ICME  =	{ICME})
@String(ICASSP=	{ICASSP})
@String(ICIP  = {ICIP})
@String(ACCV  = {ACCV})
@String(ICLR  = {ICLR})
@String(IJCAI = {IJCAI})
@String(PR = {PR})
@String(AAAI = {AAAI})
@String(CVPRW= {CVPRW})
@String(CSVT = {IEEE TCSVT})



@String(PAMI = {IEEE Trans. Pattern Anal. Mach. Intell.})
@String(IJCV = {Int. J. Comput. Vis.})
@String(CVPR= {IEEE Conf. Comput. Vis. Pattern Recog.})
@String(ICCV= {Int. Conf. Comput. Vis.})
@String(ECCV= {Eur. Conf. Comput. Vis.})
@String(NIPS= {Adv. Neural Inform. Process. Syst.})
@String(ICPR = {Int. Conf. Pattern Recog.})
@String(BMVC= {Brit. Mach. Vis. Conf.})
@String(TOG= {ACM Trans. Graph.})
@String(TIP  = {IEEE Trans. Image Process.})
@String(TVCG  = {IEEE Trans. Vis. Comput. Graph.})
@String(TMM  = {IEEE Trans. Multimedia})
@String(ACMMM= {ACM Int. Conf. Multimedia})
@String(ICME = {Int. Conf. Multimedia and Expo})
@String(ICASSP=	{ICASSP})
@String(ICIP = {IEEE Int. Conf. Image Process.})
@String(ACCV  = {ACCV})
@String(ICLR = {Int. Conf. Learn. Represent.})
@String(IJCAI = {IJCAI})
@String(PR   = {Pattern Recognition})
@String(AAAI = {AAAI})
@String(CVPRW= {IEEE Conf. Comput. Vis. Pattern Recog. Worksh.})
@String(CSVT = {IEEE Trans. Circuit Syst. Video Technol.})

@String(SPL	= {IEEE Sign. Process. Letters})
@String(VR   = {Vis. Res.})
@String(JOV	 = {J. Vis.})
@String(TVC  = {The Vis. Comput.})
@String(JCST  = {J. Comput. Sci. Tech.})
@String(CGF  = {Comput. Graph. Forum})
@String(CVM = {Computational Visual Media})


@inproceedings{kontogianni2020continuous,
	doi = {10.1007/978-3-030-58517-4_34},
	year = 2020,
	publisher = {Springer International Publishing},
	pages = {579--596},
	author = {Theodora Kontogianni and Michael Gygli and Jasper Uijlings and Vittorio Ferrari},
	title = {Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections},
	booktitle = {Computer Vision {\textendash} {ECCV} 2020}
}

@inproceedings{jang2019interactive,
	doi = {10.1109/cvpr.2019.00544},
	year = 2019,
	month = {jun},
	publisher = {{IEEE}},
	pages={5297--5306},
	author = {Won-Dong Jang and Chang-Su Kim},
	title = {Interactive Image Segmentation via Backpropagating Refinement Scheme},
	booktitle = {2019 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{sofiiuk2020f,
	doi = {10.1109/cvpr42600.2020.00865},
	year = 2020,
	month = {jun},
	publisher = {{IEEE}},
	pages={8623--8632},
	author = {Konstantin Sofiiuk and Ilia Petrov and Olga Barinova and Anton Konushin},
	title = {F-{BRS}: Rethinking Backpropagating Refinement for Interactive Segmentation},
	booktitle = {2020 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{benenson2019large,
	doi = {10.1109/cvpr.2019.01197},
	year = 2019,
	month = {jun},
	publisher = {{IEEE}},
	pages={11700--11709},
	author = {Rodrigo Benenson and Stefan Popov and Vittorio Ferrari},
	title = {Large-Scale Interactive Object Segmentation With Human Annotators},
	booktitle = {2019 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{xu2016deep,
	doi = {10.1109/cvpr.2016.47},
	year = 2016,
	month = {jun},
	publisher = {{IEEE}},
	pages={373--381},
	author = {Ning Xu and Brian Price and Scott Cohen and Jimei Yang and Thomas Huang},
	title = {Deep Interactive Object Selection},
	booktitle = {2016 {IEEE} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{agustsson2019interactive,
	doi = {10.1109/cvpr.2019.01189},
	year = 2019,
	month = {jun},
	publisher = {{IEEE}},
	pages={11622--11631},
	author = {Eirikur Agustsson and Jasper R. Uijlings and Vittorio Ferrari},
	title = {Interactive Full Image Segmentation by Considering All Regions Jointly},
	booktitle = {2019 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{bai2014error,
	doi = {10.1109/cvpr.2014.57},
	year = 2014,
	month = {jun},
	publisher = {{IEEE}},
	pages={392--399},
	author = {Junjie Bai and Xiaodong Wu},
	title = {Error-Tolerant Scribbles Based Interactive Image Segmentation},
	booktitle = {2014 {IEEE} Conference on Computer Vision and Pattern Recognition}
}

@inproceedings{acuna2018efficient,
	doi = {10.1109/cvpr.2018.00096},
	year = 2018,
	month = {jun},
	publisher = {{IEEE}},
	pages={859--868},
	author = {David Acuna and Huan Ling and Amlan Kar and Sanja Fidler},
	title = {Efficient Interactive Annotation of Segmentation Datasets with Polygon-{RNN}$++$},
	booktitle = {2018 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition}
}

@inproceedings{mahadevan2018iteratively,
  author    = {Sabarinath Mahadevan and
               Paul Voigtlaender and
               Bastian Leibe},
  title     = {Iteratively Trained Interactive Segmentation},
  booktitle = {British Machine Vision Conference 2018, {BMVC} 2018, Newcastle, UK,
               September 3-6, 2018},
  pages     = {212},
  publisher = {{BMVA} Press},
  year      = {2018}
}

@inproceedings{maninis2018deep,
	doi = {10.1109/cvpr.2018.00071},
	year = 2018,
	month = {jun},
	publisher = {{IEEE}},
	pages={616--625},
	author = {K.-K. Maninis and S. Caelles and J. Pont-Tuset and L. Van Gool},
	title = {Deep Extreme Cut: From Extreme Points to Object Segmentation},
	booktitle = {2018 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition}
}

@inproceedings{freedman2005interactive,
	doi = {10.1109/cvpr.2005.191},
	year = 2005,
	publisher = {{IEEE}},
	pages={755--762},
	author = {D. Freedman and  Tao Zhang},
	title = {Interactive Graph Cut Based Segmentation with Shape Priors},
	booktitle = {2005 {IEEE} Computer Society Conference on Computer Vision and Pattern Recognition ({CVPR}{\textquotesingle}05)}
}

@inproceedings{lin2016scribblesup,
	doi = {10.1109/cvpr.2016.344},
	year = 2016,
	month = {jun},
	publisher = {{IEEE}},
	pages={3159--3167},
	author = {Di Lin and Jifeng Dai and Jiaya Jia and Kaiming He and Jian Sun},
	title = {{ScribbleSup}: Scribble-Supervised Convolutional Networks for Semantic Segmentation},
	booktitle = {2016 {IEEE} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{lin2014microsoft,
	doi = {10.1007/978-3-319-10602-1_48},
	year = 2014,
	publisher = {Springer International Publishing},
	pages = {740--755},
	author = {Tsung-Yi Lin and Michael Maire and Serge Belongie and James Hays and Pietro Perona and Deva Ramanan and Piotr Doll{\'{a}}r and C. Lawrence Zitnick},
	title = {Microsoft {COCO}: Common Objects in Context},
	booktitle = {Computer Vision {\textendash} {ECCV} 2014}
}

@inproceedings{gupta2019lvis,
	doi = {10.1109/cvpr.2019.00550},
	year = 2019,
	month = {jun},
	publisher = {{IEEE}},
	pages={5356--5364},
	author = {Agrim Gupta and Piotr Doll{\'{a}}r and Ross Girshick},
	title = {{LVIS}: A Dataset for Large Vocabulary Instance Segmentation},
	booktitle = {2019 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{ding2020phraseclick,
	doi = {10.1007/978-3-030-58580-8_25},
	year = 2020,
	publisher = {Springer International Publishing},
	pages = {417--435},
	author = {Henghui Ding and Scott Cohen and Brian Price and Xudong Jiang},
	title = {{PhraseClick}: Toward Achieving Flexible Interactive Segmentation by Phrase and Click},
	booktitle = {Computer Vision {\textendash} {ECCV} 2020}
}

@inproceedings{lin2020interactive,
	doi = {10.1109/cvpr42600.2020.01335},
	year = 2020,
	month = {jun},
	publisher = {{IEEE}},
	pages={13339--13348},
	author = {Zheng Lin and Zhao Zhang and Lin-Zhuo Chen and Ming-Ming Cheng and Shao-Ping Lu},
	title = {Interactive Image Segmentation With First Click Attention},
	booktitle = {2020 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{liew2017regional,
	doi = {10.1109/iccv.2017.297},
	year = 2017,
	month = {oct},
	publisher = {{IEEE}},
	pages={2746--2754},
	author = {JunHao Liew and Yunchao Wei and Wei Xiong and Sim-Heng Ong and Jiashi Feng},
	title = {Regional Interactive Image Segmentation Networks},
	booktitle = {2017 {IEEE} International Conference on Computer Vision ({ICCV})}
}

@inproceedings{li2018interactive,
	doi = {10.1109/cvpr.2018.00067},
	year = 2018,
	month = {jun},
	publisher = {{IEEE}},
	pages={577--585},
	author = {Zhuwen Li and Qifeng Chen and Vladlen Koltun},
	title = {Interactive Image Segmentation with Latent Diversity},
	booktitle = {2018 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition}
}

@inproceedings{liew2019multiseg,
	doi = {10.1109/iccv.2019.00075},
	year = 2019,
	month = {oct},
	publisher = {{IEEE}},
	pages={662--670},
	author = {Jun Hao Liew and Scott Cohen and Brian Price and Long Mai and Sim-Heng Ong and Jiashi Feng},
	title = {{MultiSeg}: Semantically Meaningful, Scale-Diverse Segmentations From Minimal User Input},
	booktitle = {2019 {IEEE}/{CVF} International Conference on Computer Vision ({ICCV})}
}

@inproceedings{zhang2020interactive,
	doi = {10.1109/cvpr42600.2020.01225},
	year = 2020,
	month = {jun},
	publisher = {{IEEE}},
	pages={12234--12244},
	author = {Shiyin Zhang and Jun Hao Liew and Yunchao Wei and Shikui Wei and Yao Zhao},
	title = {Interactive Object Segmentation With Inside-Outside Guidance},
	booktitle = {2020 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@article{cheng2015densecut,
	doi = {10.1111/cgf.12758},
	year = 2015,
	month = {oct},
	publisher = {Wiley},
	volume = {34},
	number = {7},
	pages = {193--201},
	author = {M. M. Cheng and V. A. Prisacariu and S. Zheng and P. H. S. Torr and C. Rother},
	title = {{DenseCut}: Densely Connected {CRFs} for Realtime {GrabCut}},
	journal = {Computer Graphics Forum}
}

@article{rother2004grabcut,
	doi = {10.1145/1015706.1015720},
	year = 2004,
	month = {aug},
	publisher = {Association for Computing Machinery ({ACM})},
	volume = {23},
	number = {3},
	pages = {309--314},
	author = {Carsten Rother and Vladimir Kolmogorov and Andrew Blake},
	title = {\enquote{GrabCut}: interactive foreground extraction using iterated graph cuts},
	journal = {{ACM} Transactions on Graphics}
}

@inproceedings{wu2014milcut,
	doi = {10.1109/cvpr.2014.40},
	year = 2014,
	month = {jun},
	publisher = {{IEEE}},
	pages={256--263},
	author = {Jiajun Wu and Yibiao Zhao and Jun-Yan Zhu and Siwei Luo and Zhuowen Tu},
	title = {{MILCut}: A Sweeping Line Multiple Instance Learning Paradigm for Interactive Image Segmentation},
	booktitle = {2014 {IEEE} Conference on Computer Vision and Pattern Recognition}
}

@inproceedings{boykov2001interactive,
	doi = {10.1109/iccv.2001.937505},
	publisher = {{IEEE} Comput. Soc},
	volume={1},
    pages={105--112},
    year={2001},
	author = {Y.Y. Boykov and M.-P. Jolly},
	title = {Interactive graph cuts for optimal boundary {\&} region segmentation of objects in N-D images},
	booktitle = {Proceedings Eighth {IEEE} International Conference on Computer Vision. {ICCV} 2001}
}

@article{grady2006random,
	doi = {10.1109/tpami.2006.233},
	year = 2006,
	month = {nov},
	publisher = {Institute of Electrical and Electronics Engineers ({IEEE})},
	volume = {28},
	number = {11},
	pages = {1768--1783},
	author = {L. Grady},
	title = {Random Walks for Image Segmentation},
	journal = {{IEEE} Transactions on Pattern Analysis and Machine Intelligence}
}

@inproceedings{gulshan2010geodesic,
	doi = {10.1109/cvpr.2010.5540073},
	year = 2010,
	month = {jun},
	publisher = {{IEEE}},
	pages={3129--3136},
	author = {Varun Gulshan and Carsten Rother and Antonio Criminisi and Andrew Blake and Andrew Zisserman},
	title = {Geodesic star convexity for interactive image segmentation},
	booktitle = {2010 {IEEE} Computer Society Conference on Computer Vision and Pattern Recognition}
}

@inproceedings{kim2008generative,
	doi = {10.1007/978-3-540-88690-7_20},
	year = 2008,
	publisher = {Springer Berlin Heidelberg},
	pages = {264--275},
	author = {Tae Hoon Kim and Kyoung Mu Lee and Sang Uk Lee},
	title = {Generative Image Segmentation Using Random Walks with Restart},
	booktitle = {Computer Vision {\textendash} {ECCV} 2008}
}

@inproceedings{pinheiro2016learning,
	doi = {10.1007/978-3-319-46448-0_5},
	year = 2016,
	publisher = {Springer International Publishing},
	pages = {75--91},
	author = {Pedro O. Pinheiro and Tsung-Yi Lin and Ronan Collobert and Piotr Doll{\'{a}}r},
	title = {Learning to Refine Object Segments},
	booktitle = {Computer Vision {\textendash} {ECCV} 2016}
}

@inproceedings{zhou2020deepstrip,
	doi = {10.1109/cvpr42600.2020.01057},
	year = 2020,
	month = {jun},
	publisher = {{IEEE}},
	pages={10558--10567},
	author = {Peng Zhou and Brian Price and Scott Cohen and Gregg Wilensky and Larry S. Davis},
	title = {Deepstrip: High-Resolution Boundary Refinement},
	booktitle = {2020 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@incollection{yuan2020segfix,
	doi = {10.1007/978-3-030-58610-2_29},
	year = 2020,
	publisher = {Springer International Publishing},
	pages = {489--506},
	author = {Yuhui Yuan and Jingyi Xie and Xilin Chen and Jingdong Wang},
	title = {{SegFix}: Model-Agnostic Boundary Refinement for Segmentation},
	booktitle = {Computer Vision {\textendash} {ECCV} 2020}
}

@article{chen2017deeplab,
	doi = {10.1109/tpami.2017.2699184},
	year = 2017,
	month = {apr},
	publisher = {Institute of Electrical and Electronics Engineers ({IEEE})},
	volume = {40},
	number = {4},
	pages = {834--848},
	author = {Liang-Chieh Chen and George Papandreou and Iasonas Kokkinos and Kevin Murphy and Alan L. Yuille},
	title = {{DeepLab}: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected {CRFs}},
	journal = {{IEEE} Transactions on Pattern Analysis and Machine Intelligence}
}

@inproceedings{majumder2019content,
	doi = {10.1109/cvpr.2019.01187},
	year = 2019,
	month = {jun},
	publisher = {{IEEE}},
	pages={11602--11611},
	author = {Soumajit Majumder and Angela Yao},
	title = {Content-Aware Multi-Level Guidance for Interactive Instance Segmentation},
	booktitle = {2019 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{vezhnevets2005growcut,
  title={GrowCut: Interactive multi-label ND image segmentation by cellular automata},
  author={Vezhnevets, Vladimir and Konouchine, Vadim},
  booktitle={proc. of Graphicon},
  volume={1},
  number={4},
  pages={150--156},
  year={2005}
}

@inproceedings{chen2018encoder,
	doi = {10.1007/978-3-030-01234-2_49},
	year = 2018,
	publisher = {Springer International Publishing},
	pages = {833--851},
	author = {Liang-Chieh Chen and Yukun Zhu and George Papandreou and Florian Schroff and Hartwig Adam},
	title = {Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation},
	booktitle = {Computer Vision {\textendash} {ECCV} 2018}
}

@article{WangSCJDZLMTWLX19,
	doi = {10.1109/tpami.2020.2983686},
	year = 2020,
	publisher = {Institute of Electrical and Electronics Engineers ({IEEE})},
	pages = {1--1},
	author = {Jingdong Wang and Ke Sun and Tianheng Cheng and Borui Jiang and Chaorui Deng and Yang Zhao and Dong Liu and Yadong Mu and Mingkui Tan and Xinggang Wang and Wenyu Liu and Bin Xiao},
	title = {Deep High-Resolution Representation Learning for Visual Recognition},
	journal = {{IEEE} Transactions on Pattern Analysis and Machine Intelligence}
}

@inproceedings{yuan2019object,
	doi = {10.1007/978-3-030-58539-6_11},
	year = 2020,
	publisher = {Springer International Publishing},
	pages = {173--190},
	author = {Yuhui Yuan and Xilin Chen and Jingdong Wang},
	title = {Object-Contextual Representations for Semantic Segmentation},
	booktitle = {Computer Vision {\textendash} {ECCV} 2020}
}

@inproceedings{deng2009imagenet,
	doi = {10.1109/cvprw.2009.5206848},
	year = 2009,
	month = {jun},
	publisher = {{IEEE}},
    pages={248--255},
	author = {Jia Deng and  Wei Dong and R. Socher and  Li-Jia Li and  Kai Li and  Li Fei-Fei},
	title = {{ImageNet}: A large-scale hierarchical image database},
	booktitle = {2009 {IEEE} Conference on Computer Vision and Pattern Recognition}
}

@inproceedings{sofiiuk2019adaptis,
	doi = {10.1109/iccv.2019.00745},
	year = 2019,
	month = {oct},
	publisher = {{IEEE}},
	pages={7355--7363},
	author = {Konstantin Sofiiuk and Olga Barinova and Anton Konushin and Olga Barinova},
	title = {{AdaptIS}: Adaptive Instance Selection Network},
	booktitle = {2019 {IEEE}/{CVF} International Conference on Computer Vision ({ICCV})}
}

@inproceedings{lin2017focal,
	doi = {10.1109/iccv.2017.324},
	year = 2017,
	month = {oct},
	publisher = {{IEEE}},
	pages={2980--2988},
	author = {Tsung-Yi Lin and Priya Goyal and Ross Girshick and Kaiming He and Piotr Doll{\'{a}}r},
	title = {Focal Loss for Dense Object Detection},
	booktitle = {2017 {IEEE} International Conference on Computer Vision ({ICCV})}
}

@inproceedings{martin2001database,
	doi = {10.1109/iccv.2001.937655},
	publisher = {{IEEE} Comput. Soc},
	volume={2},
	pages={416--423},
	year={2001},
	author = {D. Martin and C. Fowlkes and D. Tal and J. Malik},
	title = {A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics},
	booktitle = {Proceedings Eighth {IEEE} International Conference on Computer Vision. {ICCV} 2001}
}

@inproceedings{perazzi2016benchmark,
	doi = {10.1109/cvpr.2016.85},
	year = 2016,
	month = {jun},
	publisher = {{IEEE}},
	pages={724--732},
	author = {F. Perazzi and J. Pont-Tuset and B. McWilliams and L. Van Gool and M. Gross and A. Sorkine-Hornung},
	title = {A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation},
	booktitle = {2016 {IEEE} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{hariharan2011semantic,
	doi = {10.1109/iccv.2011.6126343},
	year = 2011,
	month = {nov},
	publisher = {{IEEE}},
	pages={991--998},
	author = {Bharath Hariharan and Pablo Arbelaez and Lubomir Bourdev and Subhransu Maji and Jitendra Malik},
	title = {Semantic contours from inverse detectors},
	booktitle = {2011 International Conference on Computer Vision}
}

@article{mcguinness2010comparative,
	doi = {10.1016/j.patcog.2009.03.008},
	year = 2010,
	month = {feb},
	publisher = {Elsevier {BV}},
	volume = {43},
	number = {2},
	pages = {434--444},
	author = {Kevin McGuinness and Noel E. O'Connor},
	title = {A comparative evaluation of interactive segmentation algorithms},
	journal = {Pattern Recognition}
}

@article{everingham2010pascal,
	doi = {10.1007/s11263-009-0275-4},
	year = 2009,
	month = {sep},
	publisher = {Springer Science and Business Media {LLC}},
	volume = {88},
	number = {2},
	pages = {303--338},
	author = {Mark Everingham and Luc Van Gool and Christopher K. I. Williams and John Winn and Andrew Zisserman},
	title = {The Pascal Visual Object Classes ({VOC}) Challenge},
	journal = {International Journal of Computer Vision}
}

@article{kuznetsova2018open,
	doi = {10.1007/s11263-020-01316-z},
	year = 2020,
	month = {mar},
	publisher = {Springer Science and Business Media {LLC}},
	volume = {128},
	number = {7},
	pages = {1956--1981},
	author = {Alina Kuznetsova and Hassan Rom and Neil Alldrin and Jasper Uijlings and Ivan Krasin and Jordi Pont-Tuset and Shahab Kamali and Stefan Popov and Matteo Malloci and Alexander Kolesnikov and Tom Duerig and Vittorio Ferrari},
	title = {The Open Images Dataset V4},
	journal = {International Journal of Computer Vision}
}

@article{wang2020solov2,
  title={SOLOv2: Dynamic and Fast Instance Segmentation},
  author={Wang, Xinlong and Zhang, Rufeng and Kong, Tao and Li, Lei and Shen, Chunhua},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

@inproceedings{tan2020equalization,
	doi = {10.1109/cvpr42600.2020.01168},
	year = 2020,
	month = {jun},
	publisher = {{IEEE}},
	pages={11662--11671},
	author = {Jingru Tan and Changbao Wang and Buyu Li and Quanquan Li and Wanli Ouyang and Changqing Yin and Junjie Yan},
	title = {Equalization Loss for Long-Tailed Object Recognition},
	booktitle = {2020 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@article{jiawei2020balanced,
  title={Balanced Meta-Softmax for Long-Tailed Visual Recognition},
  author={Jiawei, Ren and Yu, Cunjun and Ma, Xiao and Zhao, Haiyu and Yi, Shuai and others},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

@inproceedings{hu2020learning,
	doi = {10.1109/cvpr42600.2020.01406},
	year = 2020,
	month = {jun},
	publisher = {{IEEE}},
	pages={14045--14054},
	author = {Xinting Hu and Yi Jiang and Kaihua Tang and Jingyuan Chen and Chunyan Miao and Hanwang Zhang},
	title = {Learning to Segment the Tail},
	booktitle = {2020 {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@incollection{rahman2016optimizing,
	doi = {10.1007/978-3-319-50835-1_22},
	year = 2016,
	publisher = {Springer International Publishing},
	pages = {234--244},
	author = {Md Atiqur Rahman and Yang Wang},
	title = {Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation},
	booktitle = {Advances in Visual Computing}
}

@inproceedings{zhou2017scene,
	doi = {10.1109/cvpr.2017.544},
	year = 2017,
	month = {jul},
	publisher = {{IEEE}},
	pages={633--641},
	author = {Bolei Zhou and Hang Zhao and Xavier Puig and Sanja Fidler and Adela Barriuso and Antonio Torralba},
	title = {Scene Parsing through {ADE}20K Dataset},
	booktitle = {2017 {IEEE} Conference on Computer Vision and Pattern Recognition ({CVPR})}
}

@inproceedings{paszke2019pytorch,
  author    = {Adam Paszke and
               Sam Gross and
               Francisco Massa and
               Adam Lerer and
               James Bradbury and
               Gregory Chanan and
               Trevor Killeen and
               Zeming Lin and
               Natalia Gimelshein and
               Luca Antiga and
               Alban Desmaison and
               Andreas K{\"{o}}pf and
               Edward Yang and
               Zachary DeVito and
               Martin Raison and
               Alykhan Tejani and
               Sasank Chilamkurthy and
               Benoit Steiner and
               Lu Fang and
               Junjie Bai and
               Soumith Chintala},
  editor    = {Hanna M. Wallach and
               Hugo Larochelle and
               Alina Beygelzimer and
               Florence d'Alch{\'{e}}{-}Buc and
               Emily B. Fox and
               Roman Garnett},
  title     = {PyTorch: An Imperative Style, High-Performance Deep Learning Library},
  booktitle = {Advances in Neural Information Processing Systems 32: Annual Conference
               on Neural Information Processing Systems 2019, NeurIPS 2019, December
               8-14, 2019, Vancouver, BC, Canada},
  pages     = {8024--8035},
  year      = {2019},
  biburl    = {https://dblp.org/rec/conf/nips/PaszkeGMLBCKLGA19.bib},
}
main.bbl
\begin{thebibliography}{51}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\DOIprefix}{doi:}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{http://dx.doi.org/#1}{\path{#1}}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Inproceedings
\bibitem[{Xu et~al.(2016)Xu, Price, Cohen, Yang, and Huang}]{xu2016deep}
\bibinfo{author}{N.~Xu}, \bibinfo{author}{B.~Price},
  \bibinfo{author}{S.~Cohen}, \bibinfo{author}{J.~Yang},
  \bibinfo{author}{T.~Huang},
\newblock \bibinfo{title}{Deep interactive object selection},
\newblock in: \bibinfo{booktitle}{2016 {IEEE} Conference on Computer Vision and
  Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2016}, pp. \bibinfo{pages}{373--381}.
  \DOIprefix\doi{10.1109/cvpr.2016.47}.
%Type = Inproceedings
\bibitem[{Jang and Kim(2019)}]{jang2019interactive}
\bibinfo{author}{W.-D. Jang}, \bibinfo{author}{C.-S. Kim},
\newblock \bibinfo{title}{Interactive image segmentation via backpropagating
  refinement scheme},
\newblock in: \bibinfo{booktitle}{2019 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2019}, pp. \bibinfo{pages}{5297--5306}.
  \DOIprefix\doi{10.1109/cvpr.2019.00544}.
%Type = Inproceedings
\bibitem[{Kontogianni et~al.(2020)Kontogianni, Gygli, Uijlings, and
  Ferrari}]{kontogianni2020continuous}
\bibinfo{author}{T.~Kontogianni}, \bibinfo{author}{M.~Gygli},
  \bibinfo{author}{J.~Uijlings}, \bibinfo{author}{V.~Ferrari},
\newblock \bibinfo{title}{Continuous adaptation for interactive object
  segmentation by learning from corrections},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2020},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2020},
  pp. \bibinfo{pages}{579--596}. \DOIprefix\doi{10.1007/978-3-030-58517-4_34}.
%Type = Inproceedings
\bibitem[{Sofiiuk et~al.(2020)Sofiiuk, Petrov, Barinova, and
  Konushin}]{sofiiuk2020f}
\bibinfo{author}{K.~Sofiiuk}, \bibinfo{author}{I.~Petrov},
  \bibinfo{author}{O.~Barinova}, \bibinfo{author}{A.~Konushin},
\newblock \bibinfo{title}{F-{BRS}: Rethinking backpropagating refinement for
  interactive segmentation},
\newblock in: \bibinfo{booktitle}{2020 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2020}, pp. \bibinfo{pages}{8623--8632}.
  \DOIprefix\doi{10.1109/cvpr42600.2020.00865}.
%Type = Inproceedings
\bibitem[{Bai and Wu(2014)}]{bai2014error}
\bibinfo{author}{J.~Bai}, \bibinfo{author}{X.~Wu},
\newblock \bibinfo{title}{Error-tolerant scribbles based interactive image
  segmentation},
\newblock in: \bibinfo{booktitle}{2014 {IEEE} Conference on Computer Vision and
  Pattern Recognition}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2014}, pp.
  \bibinfo{pages}{392--399}. \DOIprefix\doi{10.1109/cvpr.2014.57}.
%Type = Inproceedings
\bibitem[{Lin et~al.(2016)Lin, Dai, Jia, He, and Sun}]{lin2016scribblesup}
\bibinfo{author}{D.~Lin}, \bibinfo{author}{J.~Dai}, \bibinfo{author}{J.~Jia},
  \bibinfo{author}{K.~He}, \bibinfo{author}{J.~Sun},
\newblock \bibinfo{title}{{ScribbleSup}: Scribble-supervised convolutional
  networks for semantic segmentation},
\newblock in: \bibinfo{booktitle}{2016 {IEEE} Conference on Computer Vision and
  Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2016}, pp. \bibinfo{pages}{3159--3167}.
  \DOIprefix\doi{10.1109/cvpr.2016.344}.
%Type = Inproceedings
\bibitem[{Freedman and Zhang(2005)}]{freedman2005interactive}
\bibinfo{author}{D.~Freedman}, \bibinfo{author}{T.~Zhang},
\newblock \bibinfo{title}{Interactive graph cut based segmentation with shape
  priors},
\newblock in: \bibinfo{booktitle}{2005 {IEEE} Computer Society Conference on
  Computer Vision and Pattern Recognition ({CVPR}{\textquotesingle}05)},
  \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2005}, pp.
  \bibinfo{pages}{755--762}. \DOIprefix\doi{10.1109/cvpr.2005.191}.
%Type = Inproceedings
\bibitem[{Maninis et~al.(2018)Maninis, Caelles, Pont-Tuset, and
  Gool}]{maninis2018deep}
\bibinfo{author}{K.-K. Maninis}, \bibinfo{author}{S.~Caelles},
  \bibinfo{author}{J.~Pont-Tuset}, \bibinfo{author}{L.~V. Gool},
\newblock \bibinfo{title}{Deep extreme cut: From extreme points to object
  segmentation},
\newblock in: \bibinfo{booktitle}{2018 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2018}, pp. \bibinfo{pages}{616--625}.
  \DOIprefix\doi{10.1109/cvpr.2018.00071}.
%Type = Inproceedings
\bibitem[{Acuna et~al.(2018)Acuna, Ling, Kar, and Fidler}]{acuna2018efficient}
\bibinfo{author}{D.~Acuna}, \bibinfo{author}{H.~Ling},
  \bibinfo{author}{A.~Kar}, \bibinfo{author}{S.~Fidler},
\newblock \bibinfo{title}{Efficient interactive annotation of segmentation
  datasets with polygon-{RNN}$++$},
\newblock in: \bibinfo{booktitle}{2018 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2018}, pp. \bibinfo{pages}{859--868}.
  \DOIprefix\doi{10.1109/cvpr.2018.00096}.
%Type = Inproceedings
\bibitem[{Benenson et~al.(2019)Benenson, Popov, and
  Ferrari}]{benenson2019large}
\bibinfo{author}{R.~Benenson}, \bibinfo{author}{S.~Popov},
  \bibinfo{author}{V.~Ferrari},
\newblock \bibinfo{title}{Large-scale interactive object segmentation with
  human annotators},
\newblock in: \bibinfo{booktitle}{2019 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2019}, pp. \bibinfo{pages}{11700--11709}.
  \DOIprefix\doi{10.1109/cvpr.2019.01197}.
%Type = Inproceedings
\bibitem[{Agustsson et~al.(2019)Agustsson, Uijlings, and
  Ferrari}]{agustsson2019interactive}
\bibinfo{author}{E.~Agustsson}, \bibinfo{author}{J.~R. Uijlings},
  \bibinfo{author}{V.~Ferrari},
\newblock \bibinfo{title}{Interactive full image segmentation by considering
  all regions jointly},
\newblock in: \bibinfo{booktitle}{2019 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2019}, pp. \bibinfo{pages}{11622--11631}.
  \DOIprefix\doi{10.1109/cvpr.2019.01189}.
%Type = Inproceedings
\bibitem[{Lin et~al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan,
  Doll{\'{a}}r, and Zitnick}]{lin2014microsoft}
\bibinfo{author}{T.-Y. Lin}, \bibinfo{author}{M.~Maire},
  \bibinfo{author}{S.~Belongie}, \bibinfo{author}{J.~Hays},
  \bibinfo{author}{P.~Perona}, \bibinfo{author}{D.~Ramanan},
  \bibinfo{author}{P.~Doll{\'{a}}r}, \bibinfo{author}{C.~L. Zitnick},
\newblock \bibinfo{title}{Microsoft {COCO}: Common objects in context},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2014},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2014},
  pp. \bibinfo{pages}{740--755}. \DOIprefix\doi{10.1007/978-3-319-10602-1_48}.
%Type = Inproceedings
\bibitem[{Gupta et~al.(2019)Gupta, Doll{\'{a}}r, and Girshick}]{gupta2019lvis}
\bibinfo{author}{A.~Gupta}, \bibinfo{author}{P.~Doll{\'{a}}r},
  \bibinfo{author}{R.~Girshick},
\newblock \bibinfo{title}{{LVIS}: A dataset for large vocabulary instance
  segmentation},
\newblock in: \bibinfo{booktitle}{2019 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2019}, pp. \bibinfo{pages}{5356--5364}.
  \DOIprefix\doi{10.1109/cvpr.2019.00550}.
%Type = Inproceedings
\bibitem[{Mahadevan et~al.(2018)Mahadevan, Voigtlaender, and
  Leibe}]{mahadevan2018iteratively}
\bibinfo{author}{S.~Mahadevan}, \bibinfo{author}{P.~Voigtlaender},
  \bibinfo{author}{B.~Leibe},
\newblock \bibinfo{title}{Iteratively trained interactive segmentation},
\newblock in: \bibinfo{booktitle}{British Machine Vision Conference 2018,
  {BMVC} 2018, Newcastle, UK, September 3-6, 2018}, \bibinfo{publisher}{{BMVA}
  Press}, \bibinfo{year}{2018}, p. \bibinfo{pages}{212}.
%Type = Inproceedings
\bibitem[{Boykov and Jolly(2001)}]{boykov2001interactive}
\bibinfo{author}{Y.~Boykov}, \bibinfo{author}{M.-P. Jolly},
\newblock \bibinfo{title}{Interactive graph cuts for optimal boundary {\&}
  region segmentation of objects in n-d images},
\newblock in: \bibinfo{booktitle}{Proceedings Eighth {IEEE} International
  Conference on Computer Vision. {ICCV} 2001}, volume~\bibinfo{volume}{1},
  \bibinfo{publisher}{{IEEE} Comput. Soc}, \bibinfo{year}{2001}, pp.
  \bibinfo{pages}{105--112}. \DOIprefix\doi{10.1109/iccv.2001.937505}.
%Type = Article
\bibitem[{Grady(2006)}]{grady2006random}
\bibinfo{author}{L.~Grady},
\newblock \bibinfo{title}{Random walks for image segmentation},
\newblock \bibinfo{journal}{{IEEE} Transactions on Pattern Analysis and Machine
  Intelligence} \bibinfo{volume}{28} (\bibinfo{year}{2006})
  \bibinfo{pages}{1768--1783}.
%Type = Inproceedings
\bibitem[{Gulshan et~al.(2010)Gulshan, Rother, Criminisi, Blake, and
  Zisserman}]{gulshan2010geodesic}
\bibinfo{author}{V.~Gulshan}, \bibinfo{author}{C.~Rother},
  \bibinfo{author}{A.~Criminisi}, \bibinfo{author}{A.~Blake},
  \bibinfo{author}{A.~Zisserman},
\newblock \bibinfo{title}{Geodesic star convexity for interactive image
  segmentation},
\newblock in: \bibinfo{booktitle}{2010 {IEEE} Computer Society Conference on
  Computer Vision and Pattern Recognition}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2010}, pp. \bibinfo{pages}{3129--3136}.
  \DOIprefix\doi{10.1109/cvpr.2010.5540073}.
%Type = Article
\bibitem[{Rother et~al.(2004)Rother, Kolmogorov, and Blake}]{rother2004grabcut}
\bibinfo{author}{C.~Rother}, \bibinfo{author}{V.~Kolmogorov},
  \bibinfo{author}{A.~Blake},
\newblock \bibinfo{title}{\enquote{GrabCut}: interactive foreground extraction
  using iterated graph cuts},
\newblock \bibinfo{journal}{{ACM} Transactions on Graphics}
  \bibinfo{volume}{23} (\bibinfo{year}{2004}) \bibinfo{pages}{309--314}.
%Type = Inproceedings
\bibitem[{Li et~al.(2018)Li, Chen, and Koltun}]{li2018interactive}
\bibinfo{author}{Z.~Li}, \bibinfo{author}{Q.~Chen},
  \bibinfo{author}{V.~Koltun},
\newblock \bibinfo{title}{Interactive image segmentation with latent
  diversity},
\newblock in: \bibinfo{booktitle}{2018 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2018}, pp. \bibinfo{pages}{577--585}.
  \DOIprefix\doi{10.1109/cvpr.2018.00067}.
%Type = Inproceedings
\bibitem[{Liew et~al.(2017)Liew, Wei, Xiong, Ong, and Feng}]{liew2017regional}
\bibinfo{author}{J.~Liew}, \bibinfo{author}{Y.~Wei},
  \bibinfo{author}{W.~Xiong}, \bibinfo{author}{S.-H. Ong},
  \bibinfo{author}{J.~Feng},
\newblock \bibinfo{title}{Regional interactive image segmentation networks},
\newblock in: \bibinfo{booktitle}{2017 {IEEE} International Conference on
  Computer Vision ({ICCV})}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2017},
  pp. \bibinfo{pages}{2746--2754}. \DOIprefix\doi{10.1109/iccv.2017.297}.
%Type = Inproceedings
\bibitem[{Liew et~al.(2019)Liew, Cohen, Price, Mai, Ong, and
  Feng}]{liew2019multiseg}
\bibinfo{author}{J.~H. Liew}, \bibinfo{author}{S.~Cohen},
  \bibinfo{author}{B.~Price}, \bibinfo{author}{L.~Mai}, \bibinfo{author}{S.-H.
  Ong}, \bibinfo{author}{J.~Feng},
\newblock \bibinfo{title}{{MultiSeg}: Semantically meaningful, scale-diverse
  segmentations from minimal user input},
\newblock in: \bibinfo{booktitle}{2019 {IEEE}/{CVF} International Conference on
  Computer Vision ({ICCV})}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2019},
  pp. \bibinfo{pages}{662--670}. \DOIprefix\doi{10.1109/iccv.2019.00075}.
%Type = Inproceedings
\bibitem[{Lin et~al.(2020)Lin, Zhang, Chen, Cheng, and Lu}]{lin2020interactive}
\bibinfo{author}{Z.~Lin}, \bibinfo{author}{Z.~Zhang}, \bibinfo{author}{L.-Z.
  Chen}, \bibinfo{author}{M.-M. Cheng}, \bibinfo{author}{S.-P. Lu},
\newblock \bibinfo{title}{Interactive image segmentation with first click
  attention},
\newblock in: \bibinfo{booktitle}{2020 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2020}, pp. \bibinfo{pages}{13339--13348}.
  \DOIprefix\doi{10.1109/cvpr42600.2020.01335}.
%Type = Article
\bibitem[{Cheng et~al.(2015)Cheng, Prisacariu, Zheng, Torr, and
  Rother}]{cheng2015densecut}
\bibinfo{author}{M.~M. Cheng}, \bibinfo{author}{V.~A. Prisacariu},
  \bibinfo{author}{S.~Zheng}, \bibinfo{author}{P.~H.~S. Torr},
  \bibinfo{author}{C.~Rother},
\newblock \bibinfo{title}{{DenseCut}: Densely connected {CRFs} for realtime
  {GrabCut}},
\newblock \bibinfo{journal}{Computer Graphics Forum} \bibinfo{volume}{34}
  (\bibinfo{year}{2015}) \bibinfo{pages}{193--201}.
%Type = Inproceedings
\bibitem[{Wu et~al.(2014)Wu, Zhao, Zhu, Luo, and Tu}]{wu2014milcut}
\bibinfo{author}{J.~Wu}, \bibinfo{author}{Y.~Zhao}, \bibinfo{author}{J.-Y.
  Zhu}, \bibinfo{author}{S.~Luo}, \bibinfo{author}{Z.~Tu},
\newblock \bibinfo{title}{{MILCut}: A sweeping line multiple instance learning
  paradigm for interactive image segmentation},
\newblock in: \bibinfo{booktitle}{2014 {IEEE} Conference on Computer Vision and
  Pattern Recognition}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2014}, pp.
  \bibinfo{pages}{256--263}. \DOIprefix\doi{10.1109/cvpr.2014.40}.
%Type = Inproceedings
\bibitem[{Zhang et~al.(2020)Zhang, Liew, Wei, Wei, and
  Zhao}]{zhang2020interactive}
\bibinfo{author}{S.~Zhang}, \bibinfo{author}{J.~H. Liew},
  \bibinfo{author}{Y.~Wei}, \bibinfo{author}{S.~Wei},
  \bibinfo{author}{Y.~Zhao},
\newblock \bibinfo{title}{Interactive object segmentation with inside-outside
  guidance},
\newblock in: \bibinfo{booktitle}{2020 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2020}, pp. \bibinfo{pages}{12234--12244}.
  \DOIprefix\doi{10.1109/cvpr42600.2020.01225}.
%Type = Inproceedings
\bibitem[{Kim et~al.(2008)Kim, Lee, and Lee}]{kim2008generative}
\bibinfo{author}{T.~H. Kim}, \bibinfo{author}{K.~M. Lee},
  \bibinfo{author}{S.~U. Lee},
\newblock \bibinfo{title}{Generative image segmentation using random walks with
  restart},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2008},
  \bibinfo{publisher}{Springer Berlin Heidelberg}, \bibinfo{year}{2008}, pp.
  \bibinfo{pages}{264--275}. \DOIprefix\doi{10.1007/978-3-540-88690-7_20}.
%Type = Inproceedings
\bibitem[{Ding et~al.(2020)Ding, Cohen, Price, and Jiang}]{ding2020phraseclick}
\bibinfo{author}{H.~Ding}, \bibinfo{author}{S.~Cohen},
  \bibinfo{author}{B.~Price}, \bibinfo{author}{X.~Jiang},
\newblock \bibinfo{title}{{PhraseClick}: Toward achieving flexible interactive
  segmentation by phrase and click},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2020},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2020},
  pp. \bibinfo{pages}{417--435}. \DOIprefix\doi{10.1007/978-3-030-58580-8_25}.
%Type = Inproceedings
\bibitem[{Pinheiro et~al.(2016)Pinheiro, Lin, Collobert, and
  Doll{\'{a}}r}]{pinheiro2016learning}
\bibinfo{author}{P.~O. Pinheiro}, \bibinfo{author}{T.-Y. Lin},
  \bibinfo{author}{R.~Collobert}, \bibinfo{author}{P.~Doll{\'{a}}r},
\newblock \bibinfo{title}{Learning to refine object segments},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2016},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2016},
  pp. \bibinfo{pages}{75--91}. \DOIprefix\doi{10.1007/978-3-319-46448-0_5}.
%Type = Incollection
\bibitem[{Yuan et~al.(2020)Yuan, Xie, Chen, and Wang}]{yuan2020segfix}
\bibinfo{author}{Y.~Yuan}, \bibinfo{author}{J.~Xie}, \bibinfo{author}{X.~Chen},
  \bibinfo{author}{J.~Wang},
\newblock \bibinfo{title}{{SegFix}: Model-agnostic boundary refinement for
  segmentation},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2020},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2020},
  pp. \bibinfo{pages}{489--506}. \DOIprefix\doi{10.1007/978-3-030-58610-2_29}.
%Type = Article
\bibitem[{Chen et~al.(2017)Chen, Papandreou, Kokkinos, Murphy, and
  Yuille}]{chen2017deeplab}
\bibinfo{author}{L.-C. Chen}, \bibinfo{author}{G.~Papandreou},
  \bibinfo{author}{I.~Kokkinos}, \bibinfo{author}{K.~Murphy},
  \bibinfo{author}{A.~L. Yuille},
\newblock \bibinfo{title}{{DeepLab}: Semantic image segmentation with deep
  convolutional nets, atrous convolution, and fully connected {CRFs}},
\newblock \bibinfo{journal}{{IEEE} Transactions on Pattern Analysis and Machine
  Intelligence} \bibinfo{volume}{40} (\bibinfo{year}{2017})
  \bibinfo{pages}{834--848}.
%Type = Inproceedings
\bibitem[{Zhou et~al.(2020)Zhou, Price, Cohen, Wilensky, and
  Davis}]{zhou2020deepstrip}
\bibinfo{author}{P.~Zhou}, \bibinfo{author}{B.~Price},
  \bibinfo{author}{S.~Cohen}, \bibinfo{author}{G.~Wilensky},
  \bibinfo{author}{L.~S. Davis},
\newblock \bibinfo{title}{Deepstrip: High-resolution boundary refinement},
\newblock in: \bibinfo{booktitle}{2020 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2020}, pp. \bibinfo{pages}{10558--10567}.
  \DOIprefix\doi{10.1109/cvpr42600.2020.01057}.
%Type = Inproceedings
\bibitem[{Chen et~al.(2018)Chen, Zhu, Papandreou, Schroff, and
  Adam}]{chen2018encoder}
\bibinfo{author}{L.-C. Chen}, \bibinfo{author}{Y.~Zhu},
  \bibinfo{author}{G.~Papandreou}, \bibinfo{author}{F.~Schroff},
  \bibinfo{author}{H.~Adam},
\newblock \bibinfo{title}{Encoder-decoder with atrous separable convolution for
  semantic image segmentation},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2018},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2018},
  pp. \bibinfo{pages}{833--851}. \DOIprefix\doi{10.1007/978-3-030-01234-2_49}.
%Type = Article
\bibitem[{Wang et~al.(2020)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan,
  Wang, Liu, and Xiao}]{WangSCJDZLMTWLX19}
\bibinfo{author}{J.~Wang}, \bibinfo{author}{K.~Sun},
  \bibinfo{author}{T.~Cheng}, \bibinfo{author}{B.~Jiang},
  \bibinfo{author}{C.~Deng}, \bibinfo{author}{Y.~Zhao},
  \bibinfo{author}{D.~Liu}, \bibinfo{author}{Y.~Mu}, \bibinfo{author}{M.~Tan},
  \bibinfo{author}{X.~Wang}, \bibinfo{author}{W.~Liu},
  \bibinfo{author}{B.~Xiao},
\newblock \bibinfo{title}{Deep high-resolution representation learning for
  visual recognition},
\newblock \bibinfo{journal}{{IEEE} Transactions on Pattern Analysis and Machine
  Intelligence}  (\bibinfo{year}{2020}) \bibinfo{pages}{1--1}.
%Type = Inproceedings
\bibitem[{Yuan et~al.(2020)Yuan, Chen, and Wang}]{yuan2019object}
\bibinfo{author}{Y.~Yuan}, \bibinfo{author}{X.~Chen},
  \bibinfo{author}{J.~Wang},
\newblock \bibinfo{title}{Object-contextual representations for semantic
  segmentation},
\newblock in: \bibinfo{booktitle}{Computer Vision {\textendash} {ECCV} 2020},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2020},
  pp. \bibinfo{pages}{173--190}. \DOIprefix\doi{10.1007/978-3-030-58539-6_11}.
%Type = Inproceedings
\bibitem[{Deng et~al.(2009)Deng, Dong, Socher, Li, Li, and
  Fei-Fei}]{deng2009imagenet}
\bibinfo{author}{J.~Deng}, \bibinfo{author}{W.~Dong},
  \bibinfo{author}{R.~Socher}, \bibinfo{author}{L.-J. Li},
  \bibinfo{author}{K.~Li}, \bibinfo{author}{L.~Fei-Fei},
\newblock \bibinfo{title}{{ImageNet}: A large-scale hierarchical image
  database},
\newblock in: \bibinfo{booktitle}{2009 {IEEE} Conference on Computer Vision and
  Pattern Recognition}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2009}, pp.
  \bibinfo{pages}{248--255}. \DOIprefix\doi{10.1109/cvprw.2009.5206848}.
%Type = Inproceedings
\bibitem[{Majumder and Yao(2019)}]{majumder2019content}
\bibinfo{author}{S.~Majumder}, \bibinfo{author}{A.~Yao},
\newblock \bibinfo{title}{Content-aware multi-level guidance for interactive
  instance segmentation},
\newblock in: \bibinfo{booktitle}{2019 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2019}, pp. \bibinfo{pages}{11602--11611}.
  \DOIprefix\doi{10.1109/cvpr.2019.01187}.
%Type = Inproceedings
\bibitem[{Lin et~al.(2017)Lin, Goyal, Girshick, He, and
  Doll{\'{a}}r}]{lin2017focal}
\bibinfo{author}{T.-Y. Lin}, \bibinfo{author}{P.~Goyal},
  \bibinfo{author}{R.~Girshick}, \bibinfo{author}{K.~He},
  \bibinfo{author}{P.~Doll{\'{a}}r},
\newblock \bibinfo{title}{Focal loss for dense object detection},
\newblock in: \bibinfo{booktitle}{2017 {IEEE} International Conference on
  Computer Vision ({ICCV})}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2017},
  pp. \bibinfo{pages}{2980--2988}. \DOIprefix\doi{10.1109/iccv.2017.324}.
%Type = Inproceedings
\bibitem[{Sofiiuk et~al.(2019)Sofiiuk, Barinova, Konushin, and
  Barinova}]{sofiiuk2019adaptis}
\bibinfo{author}{K.~Sofiiuk}, \bibinfo{author}{O.~Barinova},
  \bibinfo{author}{A.~Konushin}, \bibinfo{author}{O.~Barinova},
\newblock \bibinfo{title}{{AdaptIS}: Adaptive instance selection network},
\newblock in: \bibinfo{booktitle}{2019 {IEEE}/{CVF} International Conference on
  Computer Vision ({ICCV})}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2019},
  pp. \bibinfo{pages}{7355--7363}. \DOIprefix\doi{10.1109/iccv.2019.00745}.
%Type = Inproceedings
\bibitem[{Hariharan et~al.(2011)Hariharan, Arbelaez, Bourdev, Maji, and
  Malik}]{hariharan2011semantic}
\bibinfo{author}{B.~Hariharan}, \bibinfo{author}{P.~Arbelaez},
  \bibinfo{author}{L.~Bourdev}, \bibinfo{author}{S.~Maji},
  \bibinfo{author}{J.~Malik},
\newblock \bibinfo{title}{Semantic contours from inverse detectors},
\newblock in: \bibinfo{booktitle}{2011 International Conference on Computer
  Vision}, \bibinfo{publisher}{{IEEE}}, \bibinfo{year}{2011}, pp.
  \bibinfo{pages}{991--998}. \DOIprefix\doi{10.1109/iccv.2011.6126343}.
%Type = Article
\bibitem[{Everingham et~al.(2009)Everingham, Gool, Williams, Winn, and
  Zisserman}]{everingham2010pascal}
\bibinfo{author}{M.~Everingham}, \bibinfo{author}{L.~V. Gool},
  \bibinfo{author}{C.~K.~I. Williams}, \bibinfo{author}{J.~Winn},
  \bibinfo{author}{A.~Zisserman},
\newblock \bibinfo{title}{The pascal visual object classes ({VOC}) challenge},
\newblock \bibinfo{journal}{International Journal of Computer Vision}
  \bibinfo{volume}{88} (\bibinfo{year}{2009}) \bibinfo{pages}{303--338}.
%Type = Article
\bibitem[{Kuznetsova et~al.(2020)Kuznetsova, Rom, Alldrin, Uijlings, Krasin,
  Pont-Tuset, Kamali, Popov, Malloci, Kolesnikov, Duerig, and
  Ferrari}]{kuznetsova2018open}
\bibinfo{author}{A.~Kuznetsova}, \bibinfo{author}{H.~Rom},
  \bibinfo{author}{N.~Alldrin}, \bibinfo{author}{J.~Uijlings},
  \bibinfo{author}{I.~Krasin}, \bibinfo{author}{J.~Pont-Tuset},
  \bibinfo{author}{S.~Kamali}, \bibinfo{author}{S.~Popov},
  \bibinfo{author}{M.~Malloci}, \bibinfo{author}{A.~Kolesnikov},
  \bibinfo{author}{T.~Duerig}, \bibinfo{author}{V.~Ferrari},
\newblock \bibinfo{title}{The open images dataset v4},
\newblock \bibinfo{journal}{International Journal of Computer Vision}
  \bibinfo{volume}{128} (\bibinfo{year}{2020}) \bibinfo{pages}{1956--1981}.
%Type = Inproceedings
\bibitem[{Hu et~al.(2020)Hu, Jiang, Tang, Chen, Miao, and
  Zhang}]{hu2020learning}
\bibinfo{author}{X.~Hu}, \bibinfo{author}{Y.~Jiang}, \bibinfo{author}{K.~Tang},
  \bibinfo{author}{J.~Chen}, \bibinfo{author}{C.~Miao},
  \bibinfo{author}{H.~Zhang},
\newblock \bibinfo{title}{Learning to segment the tail},
\newblock in: \bibinfo{booktitle}{2020 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2020}, pp. \bibinfo{pages}{14045--14054}.
  \DOIprefix\doi{10.1109/cvpr42600.2020.01406}.
%Type = Article
\bibitem[{Jiawei et~al.(2020)Jiawei, Yu, Ma, Zhao, Yi
  et~al.}]{jiawei2020balanced}
\bibinfo{author}{R.~Jiawei}, \bibinfo{author}{C.~Yu}, \bibinfo{author}{X.~Ma},
  \bibinfo{author}{H.~Zhao}, \bibinfo{author}{S.~Yi}, et~al.,
\newblock \bibinfo{title}{Balanced meta-softmax for long-tailed visual
  recognition},
\newblock \bibinfo{journal}{Advances in Neural Information Processing Systems}
  \bibinfo{volume}{33} (\bibinfo{year}{2020}).
%Type = Inproceedings
\bibitem[{Tan et~al.(2020)Tan, Wang, Li, Li, Ouyang, Yin, and
  Yan}]{tan2020equalization}
\bibinfo{author}{J.~Tan}, \bibinfo{author}{C.~Wang}, \bibinfo{author}{B.~Li},
  \bibinfo{author}{Q.~Li}, \bibinfo{author}{W.~Ouyang},
  \bibinfo{author}{C.~Yin}, \bibinfo{author}{J.~Yan},
\newblock \bibinfo{title}{Equalization loss for long-tailed object
  recognition},
\newblock in: \bibinfo{booktitle}{2020 {IEEE}/{CVF} Conference on Computer
  Vision and Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2020}, pp. \bibinfo{pages}{11662--11671}.
  \DOIprefix\doi{10.1109/cvpr42600.2020.01168}.
%Type = Article
\bibitem[{Wang et~al.(2020)Wang, Zhang, Kong, Li, and Shen}]{wang2020solov2}
\bibinfo{author}{X.~Wang}, \bibinfo{author}{R.~Zhang},
  \bibinfo{author}{T.~Kong}, \bibinfo{author}{L.~Li},
  \bibinfo{author}{C.~Shen},
\newblock \bibinfo{title}{Solov2: Dynamic and fast instance segmentation},
\newblock \bibinfo{journal}{Advances in Neural Information Processing Systems}
  \bibinfo{volume}{33} (\bibinfo{year}{2020}).
%Type = Inproceedings
\bibitem[{Martin et~al.(2001)Martin, Fowlkes, Tal, and
  Malik}]{martin2001database}
\bibinfo{author}{D.~Martin}, \bibinfo{author}{C.~Fowlkes},
  \bibinfo{author}{D.~Tal}, \bibinfo{author}{J.~Malik},
\newblock \bibinfo{title}{A database of human segmented natural images and its
  application to evaluating segmentation algorithms and measuring ecological
  statistics},
\newblock in: \bibinfo{booktitle}{Proceedings Eighth {IEEE} International
  Conference on Computer Vision. {ICCV} 2001}, volume~\bibinfo{volume}{2},
  \bibinfo{publisher}{{IEEE} Comput. Soc}, \bibinfo{year}{2001}, pp.
  \bibinfo{pages}{416--423}. \DOIprefix\doi{10.1109/iccv.2001.937655}.
%Type = Article
\bibitem[{McGuinness and O'Connor(2010)}]{mcguinness2010comparative}
\bibinfo{author}{K.~McGuinness}, \bibinfo{author}{N.~E. O'Connor},
\newblock \bibinfo{title}{A comparative evaluation of interactive segmentation
  algorithms},
\newblock \bibinfo{journal}{Pattern Recognition} \bibinfo{volume}{43}
  (\bibinfo{year}{2010}) \bibinfo{pages}{434--444}.
%Type = Inproceedings
\bibitem[{Perazzi et~al.(2016)Perazzi, Pont-Tuset, McWilliams, Gool, Gross, and
  Sorkine-Hornung}]{perazzi2016benchmark}
\bibinfo{author}{F.~Perazzi}, \bibinfo{author}{J.~Pont-Tuset},
  \bibinfo{author}{B.~McWilliams}, \bibinfo{author}{L.~V. Gool},
  \bibinfo{author}{M.~Gross}, \bibinfo{author}{A.~Sorkine-Hornung},
\newblock \bibinfo{title}{A benchmark dataset and evaluation methodology for
  video object segmentation},
\newblock in: \bibinfo{booktitle}{2016 {IEEE} Conference on Computer Vision and
  Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2016}, pp. \bibinfo{pages}{724--732}.
  \DOIprefix\doi{10.1109/cvpr.2016.85}.
%Type = Inproceedings
\bibitem[{Paszke et~al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
  Killeen, Lin, Gimelshein, Antiga, Desmaison, K{\"{o}}pf, Yang, DeVito,
  Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai, and
  Chintala}]{paszke2019pytorch}
\bibinfo{author}{A.~Paszke}, \bibinfo{author}{S.~Gross},
  \bibinfo{author}{F.~Massa}, \bibinfo{author}{A.~Lerer},
  \bibinfo{author}{J.~Bradbury}, \bibinfo{author}{G.~Chanan},
  \bibinfo{author}{T.~Killeen}, \bibinfo{author}{Z.~Lin},
  \bibinfo{author}{N.~Gimelshein}, \bibinfo{author}{L.~Antiga},
  \bibinfo{author}{A.~Desmaison}, \bibinfo{author}{A.~K{\"{o}}pf},
  \bibinfo{author}{E.~Yang}, \bibinfo{author}{Z.~DeVito},
  \bibinfo{author}{M.~Raison}, \bibinfo{author}{A.~Tejani},
  \bibinfo{author}{S.~Chilamkurthy}, \bibinfo{author}{B.~Steiner},
  \bibinfo{author}{L.~Fang}, \bibinfo{author}{J.~Bai},
  \bibinfo{author}{S.~Chintala},
\newblock \bibinfo{title}{Pytorch: An imperative style, high-performance deep
  learning library},
\newblock in: \bibinfo{editor}{H.~M. Wallach}, \bibinfo{editor}{H.~Larochelle},
  \bibinfo{editor}{A.~Beygelzimer}, \bibinfo{editor}{F.~d'Alch{\'{e}}{-}Buc},
  \bibinfo{editor}{E.~B. Fox}, \bibinfo{editor}{R.~Garnett} (Eds.),
  \bibinfo{booktitle}{Advances in Neural Information Processing Systems 32:
  Annual Conference on Neural Information Processing Systems 2019, NeurIPS
  2019, December 8-14, 2019, Vancouver, BC, Canada}, \bibinfo{year}{2019}, pp.
  \bibinfo{pages}{8024--8035}.
%Type = Inproceedings
\bibitem[{Zhou et~al.(2017)Zhou, Zhao, Puig, Fidler, Barriuso, and
  Torralba}]{zhou2017scene}
\bibinfo{author}{B.~Zhou}, \bibinfo{author}{H.~Zhao},
  \bibinfo{author}{X.~Puig}, \bibinfo{author}{S.~Fidler},
  \bibinfo{author}{A.~Barriuso}, \bibinfo{author}{A.~Torralba},
\newblock \bibinfo{title}{Scene parsing through {ADE}20k dataset},
\newblock in: \bibinfo{booktitle}{2017 {IEEE} Conference on Computer Vision and
  Pattern Recognition ({CVPR})}, \bibinfo{publisher}{{IEEE}},
  \bibinfo{year}{2017}, pp. \bibinfo{pages}{633--641}.
  \DOIprefix\doi{10.1109/cvpr.2017.544}.
%Type = Incollection
\bibitem[{Rahman and Wang(2016)}]{rahman2016optimizing}
\bibinfo{author}{M.~A. Rahman}, \bibinfo{author}{Y.~Wang},
\newblock \bibinfo{title}{Optimizing intersection-over-union in deep neural
  networks for image segmentation},
\newblock in: \bibinfo{booktitle}{Advances in Visual Computing},
  \bibinfo{publisher}{Springer International Publishing}, \bibinfo{year}{2016},
  pp. \bibinfo{pages}{234--244}. \DOIprefix\doi{10.1007/978-3-319-50835-1_22}.

\end{thebibliography}
