  An agent with an inaccurate model of its environment faces a difficult
choice: it can ignore the errors in its model and act in the real world in
whatever way it determines is optimal with respect to its model. Alternatively,
it can take a more conservative stance and eschew its model in favor of
optimizing its behavior solely via real-world interaction. This latter approach
can be exceedingly slow to learn from experience, while the former can lead to
"planner overfitting" - aspects of the agent's behavior are optimized to
exploit errors in its model. This paper explores an intermediate position in
which the planner seeks to avoid overfitting through a kind of regularization
of the plans it considers. We present three different approaches that
demonstrably mitigate planner overfitting in reinforcement-learning
environments.
