  The release of openly available, robust natural language generation
algorithms (NLG) has spurred much public attention and debate. One reason lies
in the algorithms' purported ability to generate human-like text across various
domains. Empirical evidence using incentivized tasks to assess whether people
(a) can distinguish and (b) prefer algorithm-generated versus human-written
text is lacking. We conducted two experiments assessing behavioral reactions to
the state-of-the-art Natural Language Generation algorithm GPT-2 (Ntotal =
830). Using the identical starting lines of human poems, GPT-2 produced samples
of poems. From these samples, either a random poem was chosen
(Human-out-of-the-loop) or the best one was selected (Human-in-the-loop) and in
turn matched with a human-written poem. In a new incentivized version of the
Turing Test, participants failed to reliably detect the
algorithmically-generated poems in the Human-in-the-loop treatment, yet
succeeded in the Human-out-of-the-loop treatment. Further, people reveal a
slight aversion to algorithm-generated poetry, independent on whether
participants were informed about the algorithmic origin of the poem
(Transparency) or not (Opacity). We discuss what these results convey about the
performance of NLG algorithms to produce human-like text and propose
methodologies to study such learning algorithms in human-agent experimental
settings.
