  Distributed representations of sentences have become ubiquitous in natural
language processing tasks. In this paper, we consider a continual learning
scenario for sentence representations: Given a sequence of corpora, we aim to
optimize the sentence encoder with respect to the new corpus while maintaining
its accuracy on the old corpora. To address this problem, we propose to
initialize sentence encoders with the help of corpus-independent features, and
then sequentially update sentence encoders using Boolean operations of
conceptor matrices to learn corpus-dependent features. We evaluate our approach
on semantic textual similarity tasks and show that our proposed sentence
encoder can continually learn features from new corpora while retaining its
competence on previously encountered corpora.
