  Modern deterministic retrieval pipelines prioritize achieving
state-of-the-art performance but often lack interpretability in
decision-making. These models face challenges in assessing uncertainty, leading
to overconfident predictions. To overcome these limitations, we integrate
uncertainty calibration and interpretability into a retrieval pipeline.
Specifically, we introduce Bayesian methodologies and multi-perspective
retrieval to calibrate uncertainty within a retrieval pipeline. We incorporate
techniques such as LIME and SHAP to analyze the behavior of a black-box
reranker model. The importance scores derived from these explanation
methodologies serve as supplementary relevance scores to enhance the base
reranker model. We evaluate the resulting performance enhancements achieved
through uncertainty calibration and interpretable reranking on Question
Answering and Fact Checking tasks. Our methods demonstrate substantial
performance improvements across three KILT datasets.
