  Deep Reinforcement Learning (Deep RL) has had incredible achievements on high
dimensional problems, yet its learning process remains unstable even on the
simplest tasks. Deep RL uses neural networks as function approximators. These
neural models are largely inspired by developments in the (un)supervised
machine learning community. Compared to these learning frameworks, one of the
major difficulties of RL is the absence of i.i.d. data. One way to cope with
this difficulty is to control the rate of change of the policy at every
iteration. In this work, we challenge the common practices of the
(un)supervised learning community of using a fixed neural architecture, by
having a neural model that grows in size at each policy update. This allows a
closed form entropy regularized policy update, which leads to a better control
of the rate of change of the policy at each iteration and help cope with the
non i.i.d. nature of RL. Initial experiments on classical RL benchmarks show
promising results with remarkable convergence on some RL tasks when compared to
other deep RL baselines, while exhibiting limitations on others.
