  Class-agnostic counting (CAC) has numerous potential applications across
various domains. The goal is to count objects of an arbitrary category during
testing, based on only a few annotated exemplars. In this paper, we point out
that the task of counting objects of interest when there are multiple object
classes in the image (namely, multi-class object counting) is particularly
challenging for current object counting models. They often greedily count every
object regardless of the exemplars. To address this issue, we propose
localizing the area containing the objects of interest via an exemplar-based
segmentation model before counting them. The key challenge here is the lack of
segmentation supervision to train this model. To this end, we propose a method
to obtain pseudo segmentation masks using only box exemplars and dot
annotations. We show that the segmentation model trained on these
pseudo-labeled masks can effectively localize objects of interest for an
arbitrary multi-class image based on the exemplars. To evaluate the performance
of different methods on multi-class counting, we introduce two new benchmarks,
a synthetic multi-class dataset and a new test set of real images in which
objects from multiple classes are present. Our proposed method shows a
significant advantage over the previous CAC methods on these two benchmarks.
