  Ensemble-based methods are highly popular approaches that increase the
accuracy of a decision by aggregating the opinions of individual voters. The
common point is to maximize accuracy; however, a natural limitation occurs if
incremental costs are also assigned to the individual voters. Consequently, we
investigate creating ensembles under an additional constraint on the total cost
of the members. This task can be formulated as a knapsack problem, where the
energy is the ensemble accuracy formed by some aggregation rules. However, the
generally applied aggregation rules lead to a nonseparable energy function,
which takes the common solution tools -- such as dynamic programming -- out of
action. We introduce a novel stochastic approach that considers the energy as
the joint probability function of the member accuracies. This type of knowledge
can be efficiently incorporated in a stochastic search process as a stopping
rule, since we have the information on the expected accuracy or, alternatively,
the probability of finding more accurate ensembles. Experimental analyses of
the created ensembles of pattern classifiers and object detectors confirm the
efficiency of our approach. Moreover, we propose a novel stochastic search
strategy that better fits the energy, compared with general approaches such as
simulated annealing.
