  This paper examines performance evaluation criteria for basic vision tasks
involving sets of objects namely, object detection, instance-level segmentation
and multi-object tracking. The rankings of algorithms by an existing criterion
can fluctuate with different choices of parameters, e.g. Intersection over
Union (IoU) threshold, making their evaluations unreliable. More importantly,
there is no means to verify whether we can trust the evaluations of a
criterion. This work suggests a notion of trustworthiness for performance
criteria, which requires (i) robustness to parameters for reliability, (ii)
contextual meaningfulness in sanity tests, and (iii) consistency with
mathematical requirements such as the metric properties. We observe that these
requirements were overlooked by many widely-used criteria, and explore
alternative criteria using metrics for sets of shapes. We also assess all these
criteria based on the suggested requirements for trustworthiness.
