  Scientists often use meta-analysis to characterize the impact of an
intervention on some outcome of interest across a body of literature. However,
threats to the utility and validity of meta-analytic estimates arise when
scientists average over potentially important variations in context like
different research designs. Uncertainty about quality and commensurability of
evidence casts doubt on results from meta-analysis, yet existing software tools
for meta-analysis do not necessarily emphasize addressing these concerns in
their workflows. We present MetaExplorer, a prototype system for meta-analysis
that we developed using iterative design with meta-analysis experts to provide
a guided process for eliciting assessments of uncertainty and reasoning about
how to incorporate them during statistical inference. Our qualitative
evaluation of MetaExplorer with experienced meta-analysts shows that imposing a
structured workflow both elevates the perceived importance of epistemic
concerns and presents opportunities for tools to engage users in dialogue
around goals and standards for evidence aggregation.
