  We propose bookworm continual learning(BCL), a flexible setting where unseen
classes can be inferred via a semantic model, and the visual model can be
updated continually. Thus BCL generalizes both continual learning (CL) and
zero-shot learning (ZSL). We also propose the bidirectional imagination (BImag)
framework to address BCL where features of both past and future classes are
generated. We observe that conditioning the feature generator on attributes can
actually harm the continual learning ability, and propose two variants (joint
class-attribute conditioning and asymmetric generation) to alleviate this
problem.
