  Dynamic programming is a class of algorithms used to compute optimal control
policies for Markov decision processes. Dynamic programming is ubiquitous in
control theory, and is also the foundation of reinforcement learning. In this
paper, we show that value improvement, one of the main steps of dynamic
programming, can be naturally seen as composition in a category of optics, and
intuitively, the optimal value function is the limit of a chain of optic
compositions. We illustrate this with three classic examples: the gridworld,
the inverted pendulum and the savings problem. This is a first step towards a
complete account of reinforcement learning in terms of parametrised optics.
