  Compositional generalization is an important ability of language models and
has many different manifestations. For data-to-text generation, previous
research on this ability is limited to a single manifestation called
Systematicity and lacks consideration of large language models (LLMs), which
cannot fully cover practical application scenarios. In this work, we propose
SPOR, a comprehensive and practical evaluation method for compositional
generalization in data-to-text generation. SPOR includes four aspects of
manifestations (Systematicity, Productivity, Order invariance, and Rule
learnability) and allows high-quality evaluation without additional manual
annotations based on existing datasets. We demonstrate SPOR on two different
datasets and evaluate some existing language models including LLMs. We find
that the models are deficient in various aspects of the evaluation and need
further improvement. Our work shows the necessity for comprehensive research on
different manifestations of compositional generalization in data-to-text
generation and provides a framework for evaluation.
