  Information systems increasingly leverage artificial intelligence (AI) and
machine learning (ML) to generate value from vast amounts of data. However, ML
models are imperfect and can generate incorrect classifications. Hence,
human-in-the-loop (HITL) extensions to ML models add a human review for
instances that are difficult to classify. This study argues that continuously
relying on human experts to handle difficult model classifications leads to a
strong increase in human effort, which strains limited resources. To address
this issue, we propose a hybrid system that creates artificial experts that
learn to classify data instances from unknown classes previously reviewed by
human experts. Our hybrid system assesses which artificial expert is suitable
for classifying an instance from an unknown class and automatically assigns it.
Over time, this reduces human effort and increases the efficiency of the
system. Our experiments demonstrate that our approach outperforms traditional
HITL systems for several benchmarks on image classification.
