  Has there been real progress in multi-hop question-answering? Models often
exploit dataset artifacts to produce correct answers, without connecting
information across multiple supporting facts. This limits our ability to
measure true progress and defeats the purpose of building multi-hop QA
datasets. We make three contributions towards addressing this. First, we
formalize such undesirable behavior as disconnected reasoning across subsets of
supporting facts. This allows developing a model-agnostic probe for measuring
how much any model can cheat via disconnected reasoning. Second, using a notion
of \emph{contrastive support sufficiency}, we introduce an automatic
transformation of existing datasets that reduces the amount of disconnected
reasoning. Third, our experiments suggest that there hasn't been much progress
in multi-hop QA in the reading comprehension setting. For a recent large-scale
model (XLNet), we show that only 18 points out of its answer F1 score of 72 on
HotpotQA are obtained through multifact reasoning, roughly the same as that of
a simpler RNN baseline. Our transformation substantially reduces disconnected
reasoning (19 points in answer F1). It is complementary to adversarial
approaches, yielding further reductions in conjunction.
