  Predictive models ground many state-of-the-art developments in statistical
brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers.
The principled approach to establish their validity and usefulness is
cross-validation, testing prediction on unseen data. Here, I would like to
raise awareness on error bars of cross-validation, which are often
underestimated. Simple experiments show that sample sizes of many neuroimaging
studies inherently lead to large error bars, eg $\pm$10% for 100 samples. The
standard error across folds strongly underestimates them. These large error
bars compromise the reliability of conclusions drawn with predictive models,
such as biomarkers or methods developments where, unlike with cognitive
neuroimaging MVPA approaches, more samples cannot be acquired by repeating the
experiment across many subjects. Solutions to increase sample size must be
investigated, tackling possible increases in heterogeneity of the data.
