  Generative question answering (QA) models generate answers to questions
either solely based on the parameters of the model (the closed-book setting) or
additionally retrieving relevant evidence (the open-book setting). Generative
QA models can answer some relatively complex questions, but the mechanism
through which they do so is still poorly understood. We perform several studies
aimed at better understanding the multi-hop reasoning capabilities of
generative QA models. First, we decompose multi-hop questions into multiple
corresponding single-hop questions, and find marked inconsistency in QA models'
answers on these pairs of ostensibly identical question chains. Second, we find
that models lack zero-shot multi-hop reasoning ability: when trained only on
single-hop questions, models generalize poorly to multi-hop questions. Finally,
we demonstrate that it is possible to improve models' zero-shot multi-hop
reasoning capacity through two methods that approximate real multi-hop natural
language (NL) questions by training on either concatenation of single-hop
questions or logical forms (SPARQL). In sum, these results demonstrate that
multi-hop reasoning does not emerge naturally in generative QA models, but can
be encouraged by advances in training or modeling techniques.
