  This work studies the task of glossification, of which the aim is to em
transcribe natural spoken language sentences for the Deaf (hard-of-hearing)
community to ordered sign language glosses. Previous sequence-to-sequence
language models trained with paired sentence-gloss data often fail to capture
the rich connections between the two distinct languages, leading to
unsatisfactory transcriptions. We observe that despite different grammars,
glosses effectively simplify sentences for the ease of deaf communication,
while sharing a large portion of vocabulary with sentences. This has motivated
us to implement glossification by executing a collection of editing actions,
e.g. word addition, deletion, and copying, called editing programs, on their
natural spoken language counterparts. Specifically, we design a new neural
agent that learns to synthesize and execute editing programs, conditioned on
sentence contexts and partial editing results. The agent is trained to imitate
minimal editing programs, while exploring more widely the program space via
policy gradients to optimize sequence-wise transcription quality. Results show
that our approach outperforms previous glossification models by a large margin.
