  Tokenizing raw texts into word units is an essential pre-processing step for
critical tasks in the NLP pipeline such as tagging, parsing, named entity
recognition, and more. For most languages, this tokenization step
straightforward. However, for languages with high token-internal complexity,
further token-to-word segmentation is required. Previous canonical segmentation
studies were based on character-level frameworks, with no contextualised
representation involved. Contextualized vectors a la BERT show remarkable
results in many applications, but were not shown to improve performance on
linguistic segmentation per se. Here we propose a novel neural segmentation
model which combines the best of both worlds, contextualised token
representation and char-level decoding, which is particularly effective for
languages with high token-internal complexity and extreme morphological
ambiguity. Our model shows substantial improvements in segmentation accuracy on
Hebrew and Arabic compared to the state-of-the-art, and leads to further
improvements on downstream tasks such as Part-of-Speech Tagging, Dependency
Parsing and Named-Entity Recognition, over existing pipelines. When comparing
our segmentation-first pipeline with joint segmentation and labeling in the
same settings, we show that, contrary to pre-neural studies, the pipeline
performance is superior.
