  Ensemble-based approaches are very effective in various fields in raising the
accuracy of its individual members, when some voting rule is applied for
aggregating the individual decisions. In this paper, we investigate how to find
and characterize the ensembles having the highest accuracy if the total cost of
the ensemble members is bounded. This question leads to Knapsack problem with
non-linear and non-separable objective function in binary and multiclass
classification if the majority voting is chosen for the aggregation. As the
conventional solving methods cannot be applied for this task, a novel
stochastic approach was introduced in the binary case where the energy function
is discussed as the joint probability function of the member accuracy. We show
some theoretical results with respect to the expected ensemble accuracy and its
variance in the multiclass classification problem which can help us to solve
the Knapsack problem.
