  Open Domain Question Answering requires systems to retrieve external
knowledge and perform multi-hop reasoning by composing knowledge spread over
multiple sentences. In the recently introduced open domain question answering
challenge datasets, QASC and OpenBookQA, we need to perform retrieval of facts
and compose facts to correctly answer questions. In our work, we learn a
semantic knowledge ranking model to re-rank knowledge retrieved through Lucene
based information retrieval systems. We further propose a "knowledge fusion
model" which leverages knowledge in BERT-based language models with externally
retrieved knowledge and improves the knowledge understanding of the BERT-based
language models. On both OpenBookQA and QASC datasets, the knowledge fusion
model with semantically re-ranked knowledge outperforms previous attempts.
