  In this paper we present a novel loss function, called class-agnostic
segmentation (CAS) loss. With CAS loss the class descriptors are learned during
training of the network. We don't require to define the label of a class
a-priori, rather the CAS loss clusters regions with similar appearance together
in a weakly-supervised manner. Furthermore, we show that the CAS loss function
is sparse, bounded, and robust to class-imbalance. We first apply our CAS loss
function with fully-convolutional ResNet101 and DeepLab-v3 architectures to the
binary segmentation problem of salient object detection. We investigate the
performance against the state-of-the-art methods in two settings of low and
high-fidelity training data on seven salient object detection datasets. For
low-fidelity training data (incorrect class label) class-agnostic segmentation
loss outperforms the state-of-the-art methods on salient object detection
datasets by staggering margins of around 50%. For high-fidelity training data
(correct class labels) class-agnostic segmentation models perform as good as
the state-of-the-art approaches while beating the state-of-the-art methods on
most datasets. In order to show the utility of the loss function across
different domains we then also test on general segmentation dataset, where
class-agnostic segmentation loss outperforms competing losses by huge margins.
