  Recent studies have highlighted the benefits of generating multiple synthetic
datasets for supervised learning, from increased accuracy to more effective
model selection and uncertainty estimation. These benefits have clear empirical
support, but the theoretical understanding of them is currently very light. We
seek to increase the theoretical understanding by deriving bias-variance
decompositions for several settings of using multiple synthetic datasets,
including differentially private synthetic data. Our theory predicts multiple
synthetic datasets to be especially beneficial for high-variance downstream
predictors, and yields a simple rule of thumb to select the appropriate number
of synthetic datasets in the case of mean-squared error and Brier score. We
investigate how our theory works in practice by evaluating the performance of
an ensemble over many synthetic datasets for several real datasets and
downstream predictors. The results follow our theory, showing that our insights
are practically relevant.
