  Deterministic Policy Gradient (DPG) removes a level of randomness from
standard randomized-action Policy Gradient (PG), and demonstrates substantial
empirical success for tackling complex dynamic problems involving Markov
decision processes. At the same time, though, DPG loses its ability to learn in
a model-free (i.e., actor-only) fashion, frequently necessitating the use of
critics in order to obtain consistent estimates of the associated policy-reward
gradient. In this work, we introduce Zeroth-order Deterministic Policy Gradient
(ZDPG), which approximates policy-reward gradients via two-point stochastic
evaluations of the $Q$-function, constructed by properly designed
low-dimensional action-space perturbations. Exploiting the idea of random
horizon rollouts for obtaining unbiased estimates of the $Q$-function, ZDPG
lifts the dependence on critics and restores true model-free policy learning,
while enjoying built-in and provable algorithmic stability. Additionally, we
present new finite sample complexity bounds for ZDPG, which improve upon
existing results by up to two orders of magnitude. Our findings are supported
by several numerical experiments, which showcase the effectiveness of ZDPG in a
practical setting, and its advantages over both PG and Baseline PG.
