  Not forgetting old class knowledge is a key challenge for class-incremental
learning (CIL) when the model continuously adapts to new classes. A common
technique to address this is knowledge distillation (KD), which penalizes
prediction inconsistencies between old and new models. Such prediction is made
with almost new class data, as old class data is extremely scarce due to the
strict memory limitation in CIL. In this paper, we take a deep dive into KD
losses and find that "using new class data for KD" not only hinders the model
adaption (for learning new classes) but also results in low efficiency for
preserving old class knowledge. We address this by "using the placebos of old
classes for KD", where the placebos are chosen from a free image stream, such
as Google Images, in an automatical and economical fashion. To this end, we
train an online placebo selection policy to quickly evaluate the quality of
streaming images (good or bad placebos) and use only good ones for one-time
feed-forward computation of KD. We formulate the policy training process as an
online Markov Decision Process (MDP), and introduce an online learning
algorithm to solve this MDP problem without causing much computation costs. In
experiments, we show that our method 1) is surprisingly effective even when
there is no class overlap between placebos and original old class data, 2) does
not require any additional supervision or memory budget, and 3) significantly
outperforms a number of top-performing CIL methods, in particular when using
lower memory budgets for old class exemplars, e.g., five exemplars per class.
