  Missing value imputation is a crucial preprocessing step for many machine
learning problems. However, it is often considered as a separate subtask from
downstream applications such as classification, regression, or clustering, and
thus is not optimized together with them. We hypothesize that treating the
imputation model and downstream task model together and optimizing over full
pipelines will yield better results than treating them separately. Our work
describes a novel AutoML technique for making downstream predictions with
missing data that automatically handles preprocessing, model weighting, and
selection during inference time, with minimal compute overhead. Specifically we
develop M-DEW, a Dynamic missingness-aware Ensemble Weighting (DEW) approach,
that constructs a set of two-stage imputation-prediction pipelines, trains each
component separately, and dynamically calculates a set of pipeline weights for
each sample during inference time. We thus extend previous work on dynamic
ensemble weighting to handle missing data at the level of full
imputation-prediction pipelines, improving performance and calibration on
downstream machine learning tasks over standard model averaging techniques.
M-DEW is shown to outperform the state-of-the-art in that it produces
statistically significant reductions in model perplexity in 17 out of 18
experiments, while improving average precision in 13 out of 18 experiments.
