  We introduce a novel noisy sorting model motivated by the Just Noticeable
Difference (JND) model from experimental psychology. The goal of our model is
to capture the low quality of the data that are collected from crowdsourcing
environments. Compared to other celebrated models of noisy sorting, our model
does not rely on precise data-generation assumptions and captures crowdsourced
tasks' varying levels of difficulty that can lead to different amounts of noise
in the data. To handle this challenging task, we assume we can verify some of
the collected data using expert advice. This verification procedure is costly;
hence, we aim to minimize the number of verifications we use.
  We propose a new efficient algorithm called CandidateSort, which we prove
uses the optimal number of verifications in the noisy sorting models we
consider. We characterize this optimal number of verifications by showing that
it is linear in a parameter $k$, which intuitively measures the maximum number
of comparisons that are wrong but not inconsistent in the crowdsourcing data.
