  Challenges to reproducibility and replicability have gained widespread
attention, driven by large replication projects with lukewarm success rates. A
nascent work has emerged developing algorithms to estimate the replicability of
published findings. The current study explores ways in which AI-enabled signals
of confidence in research might be integrated into the literature search. We
interview 17 PhD researchers about their current processes for literature
search and ask them to provide feedback on a replicability estimation tool. Our
findings suggest that participants tend to confuse replicability with
generalizability and related concepts. Information about replicability can
support researchers throughout the research design processes. However, the use
of AI estimation is debatable due to the lack of explainability and
transparency. The ethical implications of AI-enabled confidence assessment must
be further studied before such tools could be widely accepted. We discuss
implications for the design of technological tools to support scholarly
activities and advance replicability.
