  The development and deployment of systems using supervised machine learning
(ML) remain challenging: mainly due to the limited reliability of prediction
models and the lack of knowledge on how to effectively integrate human
intelligence into automated decision-making. Humans involvement in the ML
process is a promising and powerful paradigm to overcome the limitations of
pure automated predictions and improve the applicability of ML in practice. We
compile a catalog of design patterns to guide developers select and implement
suitable human-in-the-loop (HiL) solutions. Our catalog takes into
consideration key requirements as the cost of human involvement and model
retraining. It includes four training patterns, four deployment patterns, and
two orthogonal cooperation patterns.
