  Sanskrit Word Segmentation (SWS) is essential in making digitized texts
available and in deploying downstream tasks. It is, however, non-trivial
because of the sandhi phenomenon that modifies the characters at the word
boundaries, and needs special treatment. Existing lexicon driven approaches for
SWS make use of Sanskrit Heritage Reader, a lexicon-driven shallow parser, to
generate the complete candidate solution space, over which various methods are
applied to produce the most valid solution. However, these approaches fail
while encountering out-of-vocabulary tokens. On the other hand, purely
engineering methods for SWS have made use of recent advances in deep learning,
but cannot make use of the latent word information on availability.
  To mitigate the shortcomings of both families of approaches, we propose
Transformer based Linguistically Informed Sanskrit Tokenizer (TransLIST)
consisting of (1) a module that encodes the character input along with
latent-word information, which takes into account the sandhi phenomenon
specific to SWS and is apt to work with partial or no candidate solutions, (2)
a novel soft-masked attention to prioritize potential candidate words and (3) a
novel path ranking algorithm to rectify the corrupted predictions. Experiments
on the benchmark datasets for SWS show that TransLIST outperforms the current
state-of-the-art system by an average 7.2 points absolute gain in terms of
perfect match (PM) metric. The codebase and datasets are publicly available at
https://github.com/rsingha108/TransLIST
