  Domain adversarial adaptation in a continual setting poses a significant
challenge due to the limitations on accessing previous source domain data.
Despite extensive research in continual learning, the task of adversarial
adaptation cannot be effectively accomplished using only a small number of
stored source domain data, which is a standard setting in memory replay
approaches. This limitation arises from the erroneous empirical estimation of
$\gH$-divergence with few source domain samples. To tackle this problem, we
propose a double-head discriminator algorithm, by introducing an addition
source-only domain discriminator that are trained solely on source learning
phase. We prove that with the introduction of a pre-trained source-only domain
discriminator, the empirical estimation error of $\gH$-divergence related
adversarial loss is reduced from the source domain side. Further experiments on
existing domain adaptation benchmark show that our proposed algorithm achieves
more than 2$\%$ improvement on all categories of target domain adaptation task
while significantly mitigating the forgetting on source domain.
