  Code-Switching (CS) remains a challenge for Automatic Speech Recognition
(ASR), especially character-based models. With the combined choice of
characters from multiple languages, the outcome from character-based models
suffers from phoneme duplication, resulting in language-inconsistent spellings.
We propose Contextualized Connectionist Temporal Classification (CCTC) loss to
encourage spelling consistencies of a character-based non-autoregressive ASR
which allows for faster inference. The CCTC loss conditions the main prediction
on the predicted contexts to ensure language consistency in the spellings. In
contrast to existing CTC-based approaches, CCTC loss does not require
frame-level alignments, since the context ground truth is obtained from the
model's estimated path. Compared to the same model trained with regular CTC
loss, our method consistently improved the ASR performance on both CS and
monolingual corpora.
