  Online model-free reinforcement learning (RL) methods with continuous actions
are playing a prominent role when dealing with real-world applications such as
Robotics. However, when confronted to non-stationary environments, these
methods crucially rely on an exploration-exploitation trade-off which is rarely
dynamically and automatically adjusted to changes in the environment. Here we
propose an active exploration algorithm for RL in structured (parameterized)
continuous action space. This framework deals with a set of discrete actions,
each of which is parameterized with continuous variables. Discrete exploration
is controlled through a Boltzmann softmax function with an inverse temperature
$\beta$ parameter. In parallel, a Gaussian exploration is applied to the
continuous action parameters. We apply a meta-learning algorithm based on the
comparison between variations of short-term and long-term reward running
averages to simultaneously tune $\beta$ and the width of the Gaussian
distribution from which continuous action parameters are drawn. When applied to
a simple virtual human-robot interaction task, we show that this algorithm
outperforms continuous parameterized RL both without active exploration and
with active exploration based on uncertainty variations measured by a
Kalman-Q-learning algorithm.
