  The convergence of deterministic policy gradient under the Hadamard
parameterization is studied in the tabular setting and the linear convergence
of the algorithm is established. To this end, we first show that the error
decreases at an $O(\frac{1}{k})$ rate for all the iterations. Based on this
result, we further show that the algorithm has a faster local linear
convergence rate after $k_0$ iterations, where $k_0$ is a constant that only
depends on the MDP problem and the initialization. To show the local linear
convergence of the algorithm, we have indeed established the contraction of the
sub-optimal probability $b_s^k$ (i.e., the probability of the output policy
$\pi^k$ on non-optimal actions) when $k\ge k_0$.
