  Textual entailment models are increasingly applied in settings like
fact-checking, presupposition verification in question answering, or summary
evaluation. However, these represent a significant domain shift from existing
entailment datasets, and models underperform as a result. We propose WiCE, a
new fine-grained textual entailment dataset built on natural claim and evidence
pairs extracted from Wikipedia. In addition to standard claim-level entailment,
WiCE provides entailment judgments over sub-sentence units of the claim, and a
minimal subset of evidence sentences that support each subclaim. To support
this, we propose an automatic claim decomposition strategy using GPT-3.5 which
we show is also effective at improving entailment models' performance on
multiple datasets at test time. Finally, we show that real claims in our
dataset involve challenging verification and retrieval problems that existing
models fail to address.
