  Active policy search combines the trial-and-error methodology from policy
search with Bayesian optimization to actively find the optimal policy. First,
policy search is a type of reinforcement learning which has become very popular
for robot control, for its ability to deal with complex continuous state and
action spaces. Second, Bayesian optimization is a sample efficient global
optimization method that uses a surrogate model, like a Gaussian process, and
optimal decision making to carefully select each sample during the optimization
process. Sample efficiency is of paramount importance when each trial involves
the real robot, expensive Monte Carlo runs, or a complex simulator. Black-box
Bayesian optimization generally assumes a cost function from a stationary
process, because nonstationary modeling is usually based on prior knowledge.
However, many control problems are inherently nonstationary due to their
failure conditions, terminal states and other abrupt effects. In this paper, we
present a kernel function specially designed for Bayesian optimization, that
allows nonstationary modeling without prior knowledge, using an adaptive local
region. The new kernel results in an improved local search (exploitation),
without penalizing the global search (exploration), as shown experimentally in
well-known optimization benchmarks and robot control scenarios. We finally show
its potential for the design of the wing shape of a UAV.
