  We present a new method for probabilistic elicitation of expert knowledge
using binary responses of human experts assessing simulated data from a
statistical model, where the parameters are subject to uncertainty. The binary
responses describe either the absolute realism of individual simulations or the
relative realism of a pair of simulations in the two alternative versions of
out approach. Each version provides a nonparametric representation of the
expert belief distribution over the values of a model parameter, without
demanding the assertion of any opinion on the parameter values themselves. Our
framework also integrates the use of active learning to efficiently query the
experts, with the possibility to additionally provide a useful misspecification
diagnostic. We validate both methods on an automatic expert judging a binomial
distribution, and on human experts judging the distribution of voters across
political parties in the United States and Norway. Both methods provide
flexible and meaningful representations of the human experts' beliefs,
correctly identifying the higher dispersion of voters between parties in
Norway.
