  When designing evidence-based policies and programs, decision-makers must
distill key information from a vast and rapidly growing literature base.
Identifying relevant literature from raw search results is time and resource
intensive, and is often done by manual screening. In this study, we develop an
AI agent based on a bidirectional encoder representations from transformers
(BERT) model and incorporate it into a human team designing an evidence
synthesis product for global development. We explore the effectiveness of the
human-AI hybrid team in accelerating the evidence synthesis process. To further
improve team efficiency, we enhance the human-AI hybrid team through active
learning (AL). Specifically, we explore different sampling strategies,
including random sampling, least confidence (LC) sampling, and highest priority
(HP) sampling, to study their influence on the collaborative screening process.
Results show that incorporating the BERT-based AI agent into the human team can
reduce the human screening effort by 68.5% compared to the case of no AI
assistance and by 16.8% compared to the case of using a support vector machine
(SVM)-based AI agent for identifying 80% of all relevant documents. When we
apply the HP sampling strategy for AL, the human screening effort can be
reduced even more: by 78.3% for identifying 80% of all relevant documents
compared to no AI assistance. We apply the AL-enhanced human-AI hybrid teaming
workflow in the design process of three evidence gap maps (EGMs) for USAID and
find it to be highly effective. These findings demonstrate how AI can
accelerate the development of evidence synthesis products and promote timely
evidence-based decision making in global development in a human-AI hybrid
teaming context.
