  We study reinforcement learning (RL) for a class of continuous-time
linear-quadratic (LQ) control problems for diffusions where volatility of the
state processes depends on both state and control variables. We apply a
model-free approach that relies neither on knowledge of model parameters nor on
their estimations, and devise an actor-critic algorithm to learn the optimal
policy parameter directly. Our main contributions include the introduction of a
novel exploration schedule and a regret analysis of the proposed algorithm. We
provide the convergence rate of the policy parameter to the optimal one, and
prove that the algorithm achieves a regret bound of $O(N^{\frac{3}{4}})$ up to
a logarithmic factor. We conduct a simulation study to validate the theoretical
results and demonstrate the effectiveness and reliability of the proposed
algorithm. We also perform numerical comparisons between our method and those
of the recent model-based stochastic LQ RL studies adapted to the state- and
control-dependent volatility setting, demonstrating a better performance of the
former in terms of regret bounds.
