  In classification problems when multiples algorithms are applied to different
benchmarks a difficult issue arises, i.e., how can we rank the algorithms? In
machine learning it is common run the algorithms several times and then a
statistic is calculated in terms of means and standard deviations. In order to
compare the performance of the algorithms, it is very common to employ
statistical tests. However, these tests may also present limitations, since
they consider only the means and not the standard deviations of the obtained
results. In this paper, we present the so called A-TOPSIS, based on TOPSIS
(Technique for Order Preference by Similarity to Ideal Solution), to solve the
problem of ranking and comparing classification algorithms in terms of means
and standard deviations. We use two case studies to illustrate the A-TOPSIS for
ranking classification algorithms and the results show the suitability of
A-TOPSIS to rank the algorithms. The presented approach is general and can be
applied to compare the performance of stochastic algorithms in machine
learning. Finally, to encourage researchers to use the A-TOPSIS for ranking
algorithms we also presented in this work an easy-to-use A-TOPSIS web
framework.
