  In real-world image enhancement, it is often challenging (if not impossible)
to acquire ground-truth data, preventing the adoption of distance metrics for
objective quality assessment. As a result, one often resorts to subjective
quality assessment, the most straightforward and reliable means of evaluating
image enhancement. Conventional subjective testing requires manually
pre-selecting a small set of visual examples, which may suffer from three
sources of biases: 1) sampling bias due to the extremely sparse distribution of
the selected samples in the image space; 2) algorithmic bias due to potential
overfitting the selected samples; 3) subjective bias due to further potential
cherry-picking test results. This eventually makes the field of real-world
image enhancement more of an art than a science. Here we take steps towards
debiasing conventional subjective assessment by automatically sampling a set of
adaptive and diverse images for subsequent testing. This is achieved by casting
sample selection into a joint maximization of the discrepancy between the
enhancers and the diversity among the selected input images. Careful visual
inspection on the resulting enhanced images provides a debiased ranking of the
enhancement algorithms. We demonstrate our subjective assessment method using
three popular and practically demanding image enhancement tasks: dehazing,
super-resolution, and low-light enhancement.
