  Domain randomization is a popular method for robustly training agents to
adapt to diverse environments and real-world tasks. In this paper, we examine
how to train an agent in domain randomization environments from a nonstochastic
control perspective. We first theoretically study online control of
continuous-time linear systems under nonstochastic noises. We present a novel
two-level online algorithm, by integrating a higher-level learning strategy and
a lower-level feedback control strategy. This method offers a practical
solution, and for the first time achieves sublinear regret in continuous-time
nonstochastic systems. Compared to standard online learning algorithms, our
algorithm features a stack and skip procedure. By applying stack and skip to
the SAC (Soft Actor-Critic) algorithm, we achieved improved results in multiple
reinforcement learning tasks within domain randomization environments. Our work
provides new insights into nonasymptotic analyses of controlling
continuous-time systems. Further, our work justifies the importance of stacked
and skipped in controller learning under nonstochastic environments.
