  We introduce the first open-domain dataset, called QuaRTz, for reasoning
about textual qualitative relationships. QuaRTz contains general qualitative
statements, e.g., "A sunscreen with a higher SPF protects the skin longer.",
twinned with 3864 crowdsourced situated questions, e.g., "Billy is wearing
sunscreen with a lower SPF than Lucy. Who will be best protected from the
sun?", plus annotations of the properties being compared. Unlike previous
datasets, the general knowledge is textual and not tied to a fixed set of
relationships, and tests a system's ability to comprehend and apply textual
qualitative knowledge in a novel setting. We find state-of-the-art results are
substantially (20%) below human performance, presenting an open challenge to
the NLP community.
