  The use of large-scale vision-language datasets is limited for object
detection due to the negative impact of label noise on localization. Prior
methods have shown how such large-scale datasets can be used for pretraining,
which can provide initial signal for localization, but is insufficient without
clean bounding-box data for at least some categories. We propose a technique to
"vet" labels extracted from noisy captions, and use them for weakly-supervised
object detection (WSOD), without any bounding boxes. We analyze and annotate
the types of label noise in captions in our Caption Label Noise dataset, and
train a classifier that predicts if an extracted label is actually present in
the image or not. Our classifier generalizes across dataset boundaries and
across categories. We compare the classifier to nine baselines on five
datasets, and demonstrate that it can improve WSOD without label vetting by 30%
(31.2 to 40.5 mAP when evaluated on PASCAL VOC). See dataset at:
https://github.com/arushirai1/CLaNDataset.
