  A key component of successfully reading a passage of text is the ability to
apply knowledge gained from the passage to a new situation. In order to
facilitate progress on this kind of reading, we present ROPES, a challenging
benchmark for reading comprehension targeting Reasoning Over Paragraph Effects
in Situations. We target expository language describing causes and effects
(e.g., "animal pollinators increase efficiency of fertilization in flowers"),
as they have clear implications for new situations. A system is presented a
background passage containing at least one of these relations, a novel
situation that uses this background, and questions that require reasoning about
effects of the relationships in the background passage in the context of the
situation. We collect background passages from science textbooks and Wikipedia
that contain such phenomena, and ask crowd workers to author situations,
questions, and answers, resulting in a 14,322 question dataset. We analyze the
challenges of this task and evaluate the performance of state-of-the-art
reading comprehension models. The best model performs only slightly better than
randomly guessing an answer of the correct type, at 61.6% F1, well below the
human performance of 89.0%.
