  Machine learning applications, especially in the fields of me\-di\-cine and
social sciences, are slowly being subjected to increasing scrutiny. Similarly
to sample size planning performed in clinical and social studies, lawmakers and
funding agencies may expect statistical uncertainty estimations in machine
learning applications that impact society. In this paper, we present an
easy-to-use python package and web application for estimating prediction
confidence intervals. The package offers eight different procedures to
determine and justify the sample size and confidence of predictions from
holdout, bootstrap, cross-validation, and progressive validation experiments.
Since the package builds directly on established data analysis libraries, it
seamlessly integrates into preprocessing and exploratory data analysis steps.
Code related to this paper is available at:
https://github.com/dabrze/confidence-planner.
