  Many real-world classification problems are significantly class-imbalanced to
detriment of the class of interest. The standard set of proper evaluation
metrics is well-known but the usual assumption is that the test dataset
imbalance equals the real-world imbalance. In practice, this assumption is
often broken for various reasons. The reported results are then often too
optimistic and may lead to wrong conclusions about industrial impact and
suitability of proposed techniques. We introduce methods focusing on evaluation
under non-constant class imbalance. We show that not only the absolute values
of commonly used metrics, but even the order of classifiers in relation to the
evaluation metric used is affected by the change of the imbalance rate.
Finally, we demonstrate that using subsampling in order to get a test dataset
with class imbalance equal to the one observed in the wild is not necessary,
and eventually can lead to significant errors in classifier's performance
estimate.
