  In control problems for insect-scale direct-drive experimental platforms
under tandem wing influence, the primary challenge facing existing
reinforcement learning models is their limited safety in the exploration
process and the stability of the continuous training process. We introduce the
ConcertoRL algorithm to enhance control precision and stabilize the online
training process, which consists of two main innovations: a time-interleaved
mechanism to interweave classical controllers with reinforcement learning-based
controllers aiming to improve control precision in the initial stages, a policy
composer organizes the experience gained from previous learning to ensure the
stability of the online training process. This paper conducts a series of
experiments. First, experiments incorporating the time-interleaved mechanism
demonstrate a substantial performance boost of approximately 70% over scenarios
without reinforcement learning enhancements and a 50% increase in efficiency
compared to reference controllers with doubled control frequencies. These
results highlight the algorithm's ability to create a synergistic effect that
exceeds the sum of its parts.
