  Quantum machine learning models have been gaining significant traction within
atomistic simulation communities. Conventionally, relative model performances
are being assessed and compared using learning curves (prediction error vs.
training set size). This article illustrates the limitations of using the Mean
Absolute Error (MAE) for benchmarking, which is particularly relevant in the
case of non-normal error distributions. We analyze more specifically the
prediction error distribution of the kernel ridge regression with SLATM
representation and L 2 distance metric (KRR-SLATM-L2) for effective atomization
energies of QM7b molecules calculated at the level of theory CCSD(T)/cc-pVDZ.
Error distributions of HF and MP2 at the same basis set referenced to CCSD(T)
values were also assessed and compared to the KRR model. We show that the true
performance of the KRR-SLATM-L2 method over the QM7b dataset is poorly assessed
by the Mean Absolute Error, and can be notably improved after adaptation of the
learning set.
