  Static subword tokenization algorithms have been an essential component of
recent works on language modeling. However, their static nature results in
important flaws that degrade the models' downstream performance and robustness.
In this work, we propose MANTa, a Module for Adaptive Neural TokenizAtion.
MANTa is a differentiable tokenizer trained end-to-end with the language model.
The resulting system offers a trade-off between the expressiveness of
byte-level models and the speed of models trained using subword tokenization.
In addition, our tokenizer is highly explainable since it produces an explicit
segmentation of sequences into blocks. We evaluate our pre-trained model on
several English datasets from different domains as well as on synthetic noise.
We find that MANTa improves robustness to character perturbations and
out-of-domain data. We then show that MANTa performs comparably to other models
on the general-domain GLUE benchmark. Finally, we show that it is considerably
faster than strictly byte-level models.
