  Bias mitigators can improve algorithmic fairness in machine learning models,
but their effect on fairness is often not stable across data splits. A popular
approach to train more stable models is ensemble learning, but unfortunately,
it is unclear how to combine ensembles with mitigators to best navigate
trade-offs between fairness and predictive performance. To that end, we built
an open-source library enabling the modular composition of 8 mitigators, 4
ensembles, and their corresponding hyperparameters, and we empirically explored
the space of configurations on 13 datasets. We distilled our insights from this
exploration in the form of a guidance diagram for practitioners that we
demonstrate is robust and reproducible.
