  Indices quantifying the performance of classifiers under class-imbalance,
often suffer from distortions depending on the constitution of the test set or
the class-specific classification accuracy, creating difficulties in assessing
the merit of the classifier. We identify two fundamental conditions that a
performance index must satisfy to be respectively resilient to altering number
of testing instances from each class and the number of classes in the test set.
In light of these conditions, under the effect of class imbalance, we
theoretically analyze four indices commonly used for evaluating binary
classifiers and five popular indices for multi-class classifiers. For indices
violating any of the conditions, we also suggest remedial modification and
normalization. We further investigate the capability of the indices to retain
information about the classification performance over all the classes, even
when the classifier exhibits extreme performance on some classes. Simulation
studies are performed on high dimensional deep representations of subset of the
ImageNet dataset using four state-of-the-art classifiers tailored for handling
class imbalance. Finally, based on our theoretical findings and empirical
evidence, we recommend the appropriate indices that should be used to evaluate
the performance of classifiers in presence of class-imbalance.
