  Is it possible to build a general and automatic natural language generation
(NLG) evaluation metric? Existing learned metrics either perform
unsatisfactorily or are restricted to tasks where large human rating data is
already available. We introduce SESCORE, a model-based metric that is highly
correlated with human judgements without requiring human annotation, by
utilizing a novel, iterative error synthesis and severity scoring pipeline.
This pipeline applies a series of plausible errors to raw text and assigns
severity labels by simulating human judgements with entailment. We evaluate
SESCORE against existing metrics by comparing how their scores correlate with
human ratings. SESCORE outperforms all prior unsupervised metrics on multiple
diverse NLG tasks including machine translation, image captioning, and WebNLG
text generation. For WMT 20/21 En-De and Zh-En, SESCORE improve the average
Kendall correlation with human judgement from 0.154 to 0.195. SESCORE even
achieves comparable performance to the best supervised metric COMET, despite
receiving no human-annotated training data.
