  Performance comparison of supervised machine learning (ML) models are widely
done in terms of different confusion matrix based scores obtained on test
datasets. However, a dataset comprises several instances having different
difficulty levels. Therefore, it is more logical to compare effectiveness of ML
models on individual instances instead of comparing scores obtained for the
entire dataset. In this paper, an alternative approach is proposed for direct
comparison of supervised ML models in terms of individual instances within the
dataset. A direct comparison matrix called \emph{Prayatul Matrix} is
introduced, which accounts for comparative outcome of two ML algorithms on
different instances of a dataset. Five different performance measures are
designed based on prayatul matrix. Efficacy of the proposed approach as well as
designed measures is analyzed with four classification techniques on three
datasets. Also analyzed on four large-scale complex image datasets with four
deep learning models namely ResNet50V2, MobileNetV2, EfficientNet, and
XceptionNet. Results are evident that the newly designed measure are capable of
giving more insight about the comparing ML algorithms, which were impossible
with existing confusion matrix based scores like accuracy, precision and
recall.
