  Diarization is a crucial component in meeting transcription systems to ease
the challenges of speech enhancement and attribute the transcriptions to the
correct speaker. Particularly in the presence of overlapping or noisy speech,
these systems have problems reliably assigning the correct speaker labels,
leading to a significant amount of speaker confusion errors. We propose to add
segment-level speaker reassignment to address this issue. By revisiting, after
speech enhancement, the speaker attribution for each segment, speaker confusion
errors from the initial diarization stage are significantly reduced. Through
experiments across different system configurations and datasets, we further
demonstrate the effectiveness and applicability in various domains. Our results
show that segment-level speaker reassignment successfully rectifies at least
40% of speaker confusion word errors, highlighting its potential for enhancing
diarization accuracy in meeting transcription systems.
