  In machine learning ensembles predictions from multiple models are
aggregated. Despite widespread use and strong performance of ensembles in
applied problems little is known about the mathematical properties of
aggregating models and associated consequences for safe, explainable use of
such models. In this paper we prove a theorem that shows that any ensemble will
exhibit at least one of the following forms of prediction instability. It will
either ignore agreement among all underlying models, change its mind when none
of the underlying models have done so, or be manipulable through inclusion or
exclusion of options it would never actually predict. As a consequence,
ensemble aggregation procedures will always need to balance the benefits of
information use against the risk of these prediction instabilities. This
analysis also sheds light on what specific forms of prediction instability to
expect from particular ensemble algorithms; for example popular tree ensembles
like random forest, or xgboost will violate basic, intuitive fairness
properties. Finally, we show that this can be ameliorated by using consistent
models in asymptotic conditions.
