  Many vision tasks use secondary information at inference time -- a seed -- to
assist a computer vision model in solving a problem. For example, an initial
bounding box is needed to initialize visual object tracking. To date, all such
work makes the assumption that the seed is a good one. However, in practice,
from crowdsourcing to noisy automated seeds, this is often not the case. We
hence propose the problem of seed rejection -- determining whether to reject a
seed based on the expected performance degradation when it is provided in place
of a gold-standard seed. We provide a formal definition to this problem, and
focus on two meaningful subgoals: understanding causes of error and
understanding the model's response to noisy seeds conditioned on the primary
input. With these goals in mind, we propose a novel training method and
evaluation metrics for the seed rejection problem. We then use seeded versions
of the viewpoint estimation and fine-grained classification tasks to evaluate
these contributions. In these experiments, we show our method can reduce the
number of seeds that need to be reviewed for a target performance by over 23%
compared to strong baselines.
