  Learned representations of scientific documents can serve as valuable input
features for downstream tasks without further fine-tuning. However, existing
benchmarks for evaluating these representations fail to capture the diversity
of relevant tasks. In response, we introduce SciRepEval, the first
comprehensive benchmark for training and evaluating scientific document
representations. It includes 24 challenging and realistic tasks, 8 of which are
new, across four formats: classification, regression, ranking and search. We
then use this benchmark to study and improve the generalization ability of
scientific document representation models. We show how state-of-the-art models
like SPECTER and SciNCL struggle to generalize across the task formats, and
that simple multi-task training fails to improve them. However, a new approach
that learns multiple embeddings per document, each tailored to a different
format, can improve performance. We experiment with task-format-specific
control codes and adapters and find they outperform the existing
single-embedding state-of-the-art by over 2 points absolute. We release the
resulting family of multi-format models, called SPECTER2, for the community to
use and build on.
