  Language generation tasks that seek to mimic human ability to use language
creatively are difficult to evaluate, since one must consider creativity,
style, and other non-trivial aspects of the generated text. The goal of this
paper is to develop evaluation methods for one such task, ghostwriting of rap
lyrics, and to provide an explicit, quantifiable foundation for the goals and
future directions of this task. Ghostwriting must produce text that is similar
in style to the emulated artist, yet distinct in content. We develop a novel
evaluation methodology that addresses several complementary aspects of this
task, and illustrate how such evaluation can be used to meaningfully analyze
system performance. We provide a corpus of lyrics for 13 rap artists, annotated
for stylistic similarity, which allows us to assess the feasibility of manual
evaluation for generated verse.
