  Given the great interest in creating keyframe summaries from video, it is
surprising how little has been done to formalise their evaluation and
comparison. User studies are often carried out to demonstrate that a proposed
method generates a more appealing summary than one or two rival methods. But
larger comparison studies cannot feasibly use such user surveys. Here we
propose a discrimination capacity measure as a formal way to quantify the
improvement over the uniform baseline, assuming that one or more ground truth
summaries are available. Using the VSUMM video collection, we examine 10 video
feature types, including CNN and SURF, and 6 methods for matching frames from
two summaries. Our results indicate that a simple frame representation through
hue histograms suffices for the purposes of comparing keyframe summaries. We
subsequently propose a formal protocol for comparing summaries when ground
truth is available.
