  The advancement of transformer neural networks has significantly elevated the
capabilities of sentence similarity models, but they struggle with highly
discriminative tasks and produce sub-optimal representations of important
documents like scientific literature. With the increased reliance on retrieval
augmentation and search, representing diverse documents as concise and
descriptive vectors is crucial. This paper improves upon the vectors embeddings
of scientific literature by assembling niche datasets using co-citations as a
similarity metric, focusing on biomedical domains. We apply a novel Mixture of
Experts (MoE) extension pipeline to pretrained BERT models, where every
multi-layer perceptron section is enlarged and copied into multiple distinct
experts. Our MoE variants perform well over $N$ scientific domains with $N$
dedicated experts, whereas standard BERT models excel in only one domain.
Notably, extending just a single transformer block to MoE captures 85% of the
benefit seen from full MoE extension at every layer. This holds promise for
versatile and efficient One-Size-Fits-All transformer networks for numerically
representing diverse inputs. Our methodology marks significant advancements in
representing scientific text and holds promise for enhancing vector database
search and compilation.
