  General purpose language models (LMs) encounter difficulties when processing
domain-specific jargon and terminology, which are frequently utilized in
specialized fields such as medicine or industrial settings. Moreover, they
often find it challenging to interpret mixed speech that blends general
language with specialized jargon. This poses a challenge for automatic speech
recognition systems operating within these specific domains. In this work, we
introduce a novel approach that integrates domain-specific or secondary LM into
general-purpose LM. This strategy involves labeling, or "coloring", each word
to indicate its association with either the general or the domain-specific LM.
We develop an optimized algorithm that enhances the beam search algorithm to
effectively handle inferences involving colored words. Our evaluations indicate
that this approach is highly effective in integrating jargon into language
tasks. Notably, our method substantially lowers the error rate for
domain-specific words without compromising performance in the general domain.
