  Large language models (LLMs) bring unprecedented flexibility in defining and
executing complex, creative natural language generation (NLG) tasks. Yet, this
flexibility brings new challenges, as it introduces new degrees of freedom in
formulating the task inputs and instructions and in evaluating model
performance. To facilitate the exploration of creative NLG tasks, we propose a
three-component research framework that consists of systematic input
manipulation, reference data, and output measurement. We use this framework to
explore citation text generation -- a popular scholarly NLP task that lacks
consensus on the task definition and evaluation metric and has not yet been
tackled within the LLM paradigm. Our results highlight the importance of
systematically investigating both task instruction and input configuration when
prompting LLMs, and reveal non-trivial relationships between different
evaluation metrics used for citation text generation. Additional human
generation and human evaluation experiments provide new qualitative insights
into the task to guide future research in citation text generation. We make our
code and data publicly available.
