  In this work, we present a simple but effective method, CTCBERT, for
advancing hidden-unit BERT (HuBERT). HuBERT applies a frame-level cross-entropy
(CE) loss, which is similar to most acoustic model training. However, CTCBERT
performs the model training with the Connectionist Temporal Classification
(CTC) objective after removing duplicated IDs in each masked region. The idea
stems from the observation that there can be significant errors in alignments
when using clustered or aligned IDs. CTC learns alignments implicitly,
indicating that learning with CTC can be more flexible when misalignment
exists. We examine CTCBERT on IDs from HuBERT Iter1, HuBERT Iter2, and PBERT.
The CTC training brings consistent improvements compared to the CE training.
Furthermore, when loading blank-related parameters during finetuning, slight
improvements are observed. Evaluated on the Librispeech 960-100h setting, the
relative WER improvements of CTCBERT are 2%-11% over HuBERT and PERT on
test-other data.
