  There is a longstanding interest in capturing the error behaviour of object
detectors by finding images where their performance is likely to be
unsatisfactory. In real-world applications such as autonomous driving, it is
also crucial to characterise potential failures beyond simple requirements of
detection performance. For example, a missed detection of a pedestrian close to
an ego vehicle will generally require closer inspection than a missed detection
of a car in the distance. The problem of predicting such potential failures at
test time has largely been overlooked in the literature and conventional
approaches based on detection uncertainty fall short in that they are agnostic
to such fine-grained characterisation of errors. In this work, we propose to
reformulate the problem of finding "hard" images as a query-based hard image
retrieval task, where queries are specific definitions of "hardness", and offer
a simple and intuitive method that can solve this task for a large family of
queries. Our method is entirely post-hoc, does not require ground-truth
annotations, is independent of the choice of a detector, and relies on an
efficient Monte Carlo estimation that uses a simple stochastic model in place
of the ground-truth. We show experimentally that it can be applied successfully
to a wide variety of queries for which it can reliably identify hard images for
a given detector without any labelled data. We provide results on ranking and
classification tasks using the widely used RetinaNet, Faster-RCNN, Mask-RCNN,
and Cascade Mask-RCNN object detectors. The code for this project is available
at https://github.com/fiveai/hardest.
