  Linear dynamical systems that obey stochastic differential equations are
canonical models. While optimal control of known systems has a rich literature,
the problem is technically hard under model uncertainty and there are hardly
any results. We initiate study of this problem and aim to learn (and
simultaneously deploy) optimal actions for minimizing a quadratic cost
function. Indeed, this work is the first that comprehensively addresses the
crucial challenge of balancing exploration versus exploitation in
continuous-time systems. We present online policies that learn optimal actions
fast by carefully randomizing the parameter estimates, and establish their
performance guarantees: a regret bound that grows with square-root of time
multiplied by the number of parameters. Implementation of the policy for a
flight-control task demonstrates its efficacy. Further, we prove sharp
stability results for inexact system dynamics and tightly specify the
infinitesimal regret caused by sub-optimal actions. To obtain the results, we
conduct a novel eigenvalue-sensitivity analysis for matrix perturbation,
establish upper-bounds for comparative ratios of stochastic integrals, and
introduce the new method of policy differentiation. Our analysis sheds light on
fundamental challenges in continuous-time reinforcement learning and suggests a
useful cornerstone for similar problems.
