  We introduce the CRASS (counterfactual reasoning assessment) data set and
benchmark utilizing questionized counterfactual conditionals as a novel and
powerful tool to evaluate large language models. We present the data set design
and benchmark that supports scoring against a crowd-validated human baseline.
We test six state-of-the-art models against our benchmark. Our results show
that it poses a valid challenge for these models and opens up considerable room
for their improvement.
