  Reinforcement learning algorithms can solve dynamic decision-making and
optimal control problems. With continuous-valued state and input variables,
reinforcement learning algorithms must rely on function approximators to
represent the value function and policy mappings. Commonly used numerical
approximators, such as neural networks or basis function expansions, have two
main drawbacks: they are black-box models offering little insight into the
mappings learned, and they require extensive trial and error tuning of their
hyper-parameters. In this paper, we propose a new approach to constructing
smooth value functions in the form of analytic expressions by using symbolic
regression. We introduce three off-line methods for finding value functions
based on a state-transition model: symbolic value iteration, symbolic policy
iteration, and a direct solution of the Bellman equation. The methods are
illustrated on four nonlinear control problems: velocity control under
friction, one-link and two-link pendulum swing-up, and magnetic manipulation.
The results show that the value functions yield well-performing policies and
are compact, mathematically tractable, and easy to plug into other algorithms.
This makes them potentially suitable for further analysis of the closed-loop
system. A comparison with an alternative approach using neural networks shows
that our method outperforms the neural network-based one.
