  Machine Learning (ML) is increasingly used across many disciplines with
impressive reported results. However, recent studies suggest published
performance of ML models are often overoptimistic. Validity concerns are
underscored by findings of an inverse relationship between sample size and
reported accuracy in published ML models, contrasting with the theory of
learning curves where accuracy should improve or remain stable with increasing
sample size. This paper investigates factors contributing to overoptimism in
ML-driven science, focusing on overfitting and publication bias. We introduce a
novel stochastic model for observed accuracy, integrating parametric learning
curves and the aforementioned biases. We construct an estimator that corrects
for these biases in observed data. Theoretical and empirical results show that
our framework can estimate the underlying learning curve, providing realistic
performance assessments from published results. Applying the model to
meta-analyses of classifications of neurological conditions, we estimate the
inherent limits of ML-based prediction in each domain.
