  Reward design for reinforcement learning agents can be difficult in
situations where one not only wants the agent to achieve some effect in the
world but where one also cares about how that effect is achieved. For example,
we might wish for an agent to adhere to a tacit understanding of commonsense,
align itself to a preference for how to behave for purposes of safety, or
taking on a particular role in an interactive game. Storytelling is a mode for
communicating tacit procedural knowledge. We introduce a technique, Story
Shaping, in which a reinforcement learning agent infers tacit knowledge from an
exemplar story of how to accomplish a task and intrinsically rewards itself for
performing actions that make its current environment adhere to that of the
inferred story world. Specifically, Story Shaping infers a knowledge graph
representation of the world state from observations, and also infers a
knowledge graph from the exemplar story. An intrinsic reward is generated based
on the similarity between the agent's inferred world state graph and the
inferred story world graph. We conducted experiments in text-based games
requiring commonsense reasoning and shaping the behaviors of agents as virtual
game characters.
