  The growth of domain-specific applications of semantic models, boosted by the
recent achievements of unsupervised embedding learning algorithms, demands
domain-specific evaluation datasets. In many cases, content-based recommenders
being a prime example, these models are required to rank words or texts
according to their semantic relatedness to a given concept, with particular
focus on top ranks. In this work, we give a threefold contribution to address
these requirements: (i) we define a protocol for the construction, based on
adaptive pairwise comparisons, of a relatedness-based evaluation dataset
tailored on the available resources and optimized to be particularly accurate
in top-rank evaluation; (ii) we define appropriate metrics, extensions of
well-known ranking correlation coefficients, to evaluate a semantic model via
the aforementioned dataset by taking into account the greater significance of
top ranks. Finally, (iii) we define a stochastic transitivity model to simulate
semantic-driven pairwise comparisons, which confirms the effectiveness of the
proposed dataset construction protocol.
