  Understanding the strengths and weaknesses of machine learning (ML)
algorithms is crucial for determine their scope of application. Here, we
introduce the DIverse and GENerative ML Benchmark (DIGEN) - a collection of
synthetic datasets for comprehensive, reproducible, and interpretable
benchmarking of machine learning algorithms for classification of binary
outcomes. The DIGEN resource consists of 40 mathematical functions which map
continuous features to discrete endpoints for creating synthetic datasets.
These 40 functions were discovered using a heuristic algorithm designed to
maximize the diversity of performance among multiple popular machine learning
algorithms thus providing a useful test suite for evaluating and comparing new
methods. Access to the generative functions facilitates understanding of why a
method performs poorly compared to other algorithms thus providing ideas for
improvement. The resource with extensive documentation and analyses is
open-source and available on GitHub.
