  Numerous evaluation metrics have been developed for natural language
generation tasks, but their effectiveness in evaluating stories is limited as
they are not specifically tailored to assess intricate aspects of storytelling,
such as fluency and interestingness. In this paper, we introduce DELTASCORE, a
novel methodology that employs perturbation techniques for the evaluation of
nuanced story aspects. Our central proposition posits that the extent to which
a story excels in a specific aspect (e.g., fluency) correlates with the
magnitude of its susceptibility to particular perturbations (e.g., the
introduction of typos). Given this, we measure the quality of an aspect by
calculating the likelihood difference between pre- and post-perturbation states
using pre-trained language models. We compare DELTASCORE with existing metrics
on storytelling datasets from two domains in five fine-grained story aspects:
fluency, coherence, relatedness, logicality, and interestingness. DELTASCORE
demonstrates remarkable performance, revealing a surprising finding that a
specific perturbation proves highly effective in capturing multiple aspects.
