  Quantum computing has promised significant improvement in solving difficult
computational tasks over classical computers. Designing quantum circuits for
practical use, however, is not a trivial objective and requires expert-level
knowledge. To aid this endeavor, this paper proposes a machine learning-based
method to construct quantum circuit architectures. Previous works have
demonstrated that classical deep reinforcement learning (DRL) algorithms can
successfully construct quantum circuit architectures without encoded physics
knowledge. However, these DRL-based works are not generalizable to settings
with changing device noises, thus requiring considerable amounts of training
resources to keep the RL models up-to-date. With this in mind, we incorporated
continual learning to enhance the performance of our algorithm. In this paper,
we present the Probabilistic Policy Reuse with deep Q-learning (PPR-DQL)
framework to tackle this circuit design challenge. By conducting numerical
simulations over various noise patterns, we demonstrate that the RL agent with
PPR was able to find the quantum gate sequence to generate the two-qubit Bell
state faster than the agent that was trained from scratch. The proposed
framework is general and can be applied to other quantum gate synthesis or
control problems -- including the automatic calibration of quantum devices.
