  We present a new pre-trained language model (PLM) for modern Hebrew, termed
AlephBERTGimmel, which employs a much larger vocabulary (128K items) than
standard Hebrew PLMs before. We perform a contrastive analysis of this model
against all previous Hebrew PLMs (mBERT, heBERT, AlephBERT) and assess the
effects of larger vocabularies on task performance. Our experiments show that
larger vocabularies lead to fewer splits, and that reducing splits is better
for model performance, across different tasks. All in all this new model
achieves new SOTA on all available Hebrew benchmarks, including Morphological
Segmentation, POS Tagging, Full Morphological Analysis, NER, and Sentiment
Analysis. Subsequently we advocate for PLMs that are larger not only in terms
of number of layers or training data, but also in terms of their vocabulary. We
release the new model publicly for unrestricted use.
