  Many useful tasks on scientific documents, such as topic classification and
citation prediction, involve corpora that span multiple scientific domains.
Typically, such tasks are accomplished by representing the text with a vector
embedding obtained from a Transformer's single CLS token. In this paper, we
argue that using multiple CLS tokens could make a Transformer better specialize
to multiple scientific domains. We present Multi2SPE: it encourages each of
multiple CLS tokens to learn diverse ways of aggregating token embeddings, then
sums them up together to create a single vector representation. We also propose
our new multi-domain benchmark, Multi-SciDocs, to test scientific paper vector
encoders under multi-domain settings. We show that Multi2SPE reduces error by
up to 25 percent in multi-domain citation prediction, while requiring only a
negligible amount of computation in addition to one BERT forward pass.
