  Open book question answering is a subset of question answering tasks where
the system aims to find answers in a given set of documents (open-book) and
common knowledge about a topic. This article proposes a solution for answering
natural language questions from a corpus of Amazon Web Services (AWS) technical
documents with no domain-specific labeled data (zero-shot). These questions can
have yes-no-none answers, short answers, long answers, or any combination of
the above. This solution comprises a two-step architecture in which a retriever
finds the right document and an extractor finds the answers in the retrieved
document. We are introducing a new test dataset for open-book QA based on real
customer questions on AWS technical documentation. After experimenting with
several information retrieval systems and extractor models based on extractive
language models, the solution attempts to find the yes-no-none answers and text
answers in the same pass. The model is trained on the The Stanford Question
Answering Dataset - SQuAD (Rajpurkaret al., 2016) and Natural Questions
(Kwiatkowski et al., 2019) datasets. We were able to achieve 49% F1 and 39%
exact match score (EM) end-to-end with no domain-specific training.
