  Despite notable advancements in automatic speech recognition (ASR),
performance tends to degrade when faced with adverse conditions. Generative
error correction (GER) leverages the exceptional text comprehension
capabilities of large language models (LLM), delivering impressive performance
in ASR error correction, where N-best hypotheses provide valuable information
for transcription prediction. However, GER encounters challenges such as fixed
N-best hypotheses, insufficient utilization of acoustic information, and
limited specificity to multi-accent scenarios. In this paper, we explore the
application of GER in multi-accent scenarios. Accents represent deviations from
standard pronunciation norms, and the multi-task learning framework for
simultaneous ASR and accent recognition (AR) has effectively addressed the
multi-accent scenarios, making it a prominent solution. In this work, we
propose a unified ASR-AR GER model, named MMGER, leveraging multi-modal
correction, and multi-granularity correction. Multi-task ASR-AR learning is
employed to provide dynamic 1-best hypotheses and accent embeddings.
Multi-modal correction accomplishes fine-grained frame-level correction by
force-aligning the acoustic features of speech with the corresponding
character-level 1-best hypothesis sequence. Multi-granularity correction
supplements the global linguistic information by incorporating regular 1-best
hypotheses atop fine-grained multi-modal correction to achieve coarse-grained
utterance-level correction. MMGER effectively mitigates the limitations of GER
and tailors LLM-based ASR error correction for the multi-accent scenarios.
Experiments conducted on the multi-accent Mandarin KeSpeech dataset demonstrate
the efficacy of MMGER, achieving a 26.72% relative improvement in AR accuracy
and a 27.55% relative reduction in ASR character error rate, compared to a
well-established standard baseline.
