  Recent studies show that models trained on synthetic datasets are able to
achieve better generalizable person re-identification (GPReID) performance than
that trained on public real-world datasets. On the other hand, due to the
limitations of real-world person ReID datasets, it would also be important and
interesting to use large-scale synthetic datasets as test sets to benchmark
person ReID algorithms. Yet this raises a critical question: is synthetic
dataset reliable for benchmarking generalizable person re-identification? In
the literature there is no evidence showing this. To address this, we design a
method called Pairwise Ranking Analysis (PRA) to quantitatively measure the
ranking similarity and perform the statistical test of identical distributions.
Specifically, we employ Kendall rank correlation coefficients to evaluate
pairwise similarity values between algorithm rankings on different datasets.
Then, a non-parametric two-sample Kolmogorov-Smirnov (KS) test is performed for
the judgement of whether algorithm ranking correlations between synthetic and
real-world datasets and those only between real-world datasets lie in identical
distributions. We conduct comprehensive experiments, with ten representative
algorithms, three popular real-world person ReID datasets, and three recently
released large-scale synthetic datasets. Through the designed pairwise ranking
analysis and comprehensive evaluations, we conclude that a recent large-scale
synthetic dataset ClonedPerson can be reliably used to benchmark GPReID,
statistically the same as real-world datasets. Therefore, this study guarantees
the usage of synthetic datasets for both source training set and target testing
set, with completely no privacy concerns from real-world surveillance data.
Besides, the study in this paper might also inspire future designs of synthetic
datasets.
