  Performance of speaker recognition systems is evaluated on test trials.
Although as crucial as rulers for tailors, trials have not been carefully
treated so far, and most existing benchmarks compose trials by naive
cross-pairing. In this paper, we argue that the cross-pairing approach produces
overwhelming easy trials, which in turn leads to potential bias in system and
technique comparison. To solve the problem, we advocate more attention to hard
trials. We present an SVM-based approach to identifying hard trials and use it
to construct new evaluation sets for VoxCeleb1 and SITW. With the new sets, we
can re-evaluate the contribution of some recent technologies. The code and the
identified hard trials will be published online at http://project.cslt.org.
