  The advent of large pre-trained generative language models has provided a
common framework for AI story generation via sampling the model to create
sequences that continue the story. However, sampling alone is insufficient for
story generation. In particular, it is hard to direct a language model to
create stories to reach a specific goal event. We present two automated
techniques grounded in deep reinforcement learning and reward shaping to
control the plot of computer-generated stories. The first utilizes proximal
policy optimization to fine-tune an existing transformer-based language model
to generate text continuations but also be goal-seeking. The second extracts a
knowledge graph from the unfolding story, which is used by a policy network
with graph attention to select a candidate continuation generated by a language
model. We report on automated metrics pertaining to how often stories achieve a
given goal event as well as human participant rankings of coherence and overall
story quality compared to baselines and ablations.
