  Evidence retrieval is a key component of explainable question answering (QA).
We argue that, despite recent progress, transformer network-based approaches
such as universal sentence encoder (USE-QA) do not always outperform
traditional information retrieval (IR) methods such as BM25 for evidence
retrieval for QA. We introduce a lexical probing task that validates this
observation: we demonstrate that neural IR methods have the capacity to capture
lexical differences between questions and answers, but miss obvious lexical
overlap signal. Learning from this probing analysis, we introduce a hybrid
approach for evidence retrieval that combines the advantages of both IR
directions. Our approach uses a routing classifier that learns when to direct
incoming questions to BM25 vs. USE-QA for evidence retrieval using very simple
statistics, which can be efficiently extracted from the top candidate evidence
sentences produced by a BM25 model. We demonstrate that this hybrid evidence
retrieval generally performs better than either individual retrieval strategy
on three QA datasets: OpenBookQA, ReQA SQuAD, and ReQA NQ. Furthermore, we show
that the proposed routing strategy is considerably faster than neural methods,
with a runtime that is up to 5 times faster than USE-QA.
