Comparing Efficiency of Expert Data Aggregation Methods
  Expert estimation of objects takes place when there are no benchmark values
of object weights, but these weights still have to be defined. That is why it
is problematic to define the efficiency of expert estimation methods. We
propose to define efficiency of such methods based on stability of their
results under perturbations of input data. We compare two modifications of
combinatorial method of expert data aggregation (spanning tree enumeration).
Using the example of these two methods, we illustrate two approaches to
efficiency evaluation. The first approach is based on usage of real data,
obtained through estimation of a set of model objects by a group of experts.
The second approach is based on simulation of the whole expert examination
cycle (including expert estimates). During evaluation of efficiency of the two
listed modifications of combinatorial expert data aggregation method the
simulation-based approach proved more robust and credible. Our experimental
study confirms that if weights of spanning trees are taken into consideration,
the results of combinatorial data aggregation method become more stable. So,
weighted spanning tree enumeration method has an advantage over non-weighted
method (and, consequently, over logarithmic least squares and row geometric
mean methods).
