  Recent advances in deep learning have enabled optimization of deep reactive
policies (DRPs) for continuous MDP planning by encoding a parametric policy as
a deep neural network and exploiting automatic differentiation in an end-to-end
model-based gradient descent framework. This approach has proven effective for
optimizing DRPs in nonlinear continuous MDPs, but it requires a large number of
sampled trajectories to learn effectively and can suffer from high variance in
solution quality. In this work, we revisit the overall model-based DRP
objective and instead take a minorization-maximization perspective to
iteratively optimize the DRP w.r.t. a locally tight lower-bounded objective.
This novel formulation of DRP learning as iterative lower bound optimization
(ILBO) is particularly appealing because (i) each step is structurally easier
to optimize than the overall objective, (ii) it guarantees a monotonically
improving objective under certain theoretical conditions, and (iii) it reuses
samples between iterations thus lowering sample complexity. Empirical
evaluation confirms that ILBO is significantly more sample-efficient than the
state-of-the-art DRP planner and consistently produces better solution quality
with lower variance. We additionally demonstrate that ILBO generalizes well to
new problem instances (i.e., different initial states) without requiring
retraining.
