  Question Answering for complex questions is often modeled as a graph
construction or traversal task, where a solver must build or traverse a graph
of facts that answer and explain a given question. This "multi-hop" inference
has been shown to be extremely challenging, with few models able to aggregate
more than two facts before being overwhelmed by "semantic drift", or the
tendency for long chains of facts to quickly drift off topic. This is a major
barrier to current inference models, as even elementary science questions
require an average of 4 to 6 facts to answer and explain. In this work we
empirically characterize the difficulty of building or traversing a graph of
sentences connected by lexical overlap, by evaluating chance sentence
aggregation quality through 9,784 manually-annotated judgments across knowledge
graphs built from three free-text corpora (including study guides and Simple
Wikipedia). We demonstrate semantic drift tends to be high and aggregation
quality low, at between 0.04% and 3%, and highlight scenarios that maximize the
likelihood of meaningfully combining information.
