  This paper studies the convergence of the mirror descent algorithm for finite
horizon stochastic control problems with measure-valued control processes. The
control objective involves a convex regularisation function, denoted as $h$,
with regularisation strength determined by the weight $\tau\ge 0$. The setting
covers regularised relaxed control problems. Under suitable conditions, we
establish the relative smoothness and convexity of the control objective with
respect to the Bregman divergence of $h$, and prove linear convergence of the
algorithm for $\tau=0$ and exponential convergence for $\tau>0$. The results
apply to common regularisers including relative entropy, $\chi^2$-divergence,
and entropic Wasserstein costs. This validates recent reinforcement learning
heuristics that adding regularisation accelerates the convergence of gradient
methods. The proof exploits careful regularity estimates of backward stochastic
differential equations in the bounded mean oscillation norm.
