  In biospectroscopy, suitably annotated and statistically independent samples
(e. g. patients, batches, etc.) for classifier training and testing are scarce
and costly. Learning curves show the model performance as function of the
training sample size and can help to determine the sample size needed to train
good classifiers. However, building a good model is actually not enough: the
performance must also be proven. We discuss learning curves for typical small
sample size situations with 5 - 25 independent samples per class. Although the
classification models achieve acceptable performance, the learning curve can be
completely masked by the random testing uncertainty due to the equally limited
test sample size. In consequence, we determine test sample sizes necessary to
achieve reasonable precision in the validation and find that 75 - 100 samples
will usually be needed to test a good but not perfect classifier. Such a data
set will then allow refined sample size planning on the basis of the achieved
performance. We also demonstrate how to calculate necessary sample sizes in
order to show the superiority of one classifier over another: this often
requires hundreds of statistically independent test samples or is even
theoretically impossible. We demonstrate our findings with a data set of ca.
2550 Raman spectra of single cells (five classes: erythrocytes, leukocytes and
three tumour cell lines BT-20, MCF-7 and OCI-AML3) as well as by an extensive
simulation that allows precise determination of the actual performance of the
models in question.
