  It has been established that training a box-based detector network can
enhance the localization performance of weakly supervised and unsupervised
methods. Moreover, we extend this understanding by demonstrating that these
detectors can be utilized to improve the original network, paving the way for
further advancements. To accomplish this, we train the detectors on top of the
network output instead of the image data and apply suitable loss
backpropagation. Our findings reveal a significant improvement in phrase
grounding for the ``what is where by looking'' task, as well as various methods
of unsupervised object discovery. Our code is available at
https://github.com/eyalgomel/box-based-refinement.
