  In this paper, we propose a novel stroke constrained attention network (SCAN)
which treats stroke as the basic unit for encoder-decoder based online
handwritten mathematical expression recognition (HMER). Unlike previous methods
which use trace points or image pixels as basic units, SCAN makes full use of
stroke-level information for better alignment and representation. The proposed
SCAN can be adopted in both single-modal (online or offline) and multi-modal
HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract
point-level features from input traces in online mode and employs a CNN encoder
to extract pixel-level features from input images in offline mode, then use
stroke constrained information to convert them into online and offline
stroke-level features. Using stroke-level features can explicitly group points
or pixels belonging to the same stroke, therefore reduces the difficulty of
symbol segmentation and recognition via the decoder with attention mechanism.
For multi-modal HMER, other than fusing multi-modal information in decoder,
SCAN can also fuse multi-modal information in encoder by utilizing the stroke
based alignments between online and offline modalities. The encoder fusion is a
better way for combining multi-modal information as it implements the
information interaction one step before the decoder fusion so that the
advantages of multiple modalities can be exploited earlier and more adequately
when training the encoder-decoder model. Evaluated on a benchmark published by
CROHME competition, the proposed SCAN achieves the state-of-the-art
performance.
