  We study the problem of data-driven, constrained control of unknown nonlinear
dynamics from a single ongoing and finite-horizon trajectory. We consider a
one-step optimal control problem with a smooth, black-box objective, typically
a composition of a known cost function and the unknown dynamics. We investigate
an on-the-fly control paradigm, i.e., at each time step, the evolution of the
dynamics and the first-order information of the cost are provided only for the
executed control action. We propose an optimization-based control algorithm
that iteratively minimizes a data-driven surrogate function for the unknown
objective. We prove that the proposed approach incurs sublinear cumulative
regret (step-wise suboptimality with respect to an optimal one-step controller)
and is worst-case optimal among a broad class of data-driven control
algorithms. We also present tractable reformulations of the approach that can
leverage off-the-shelf solvers for efficient implementations.
