  Existing auto-regressive language models have demonstrated a remarkable
capability to perform a new task with just a few examples in prompt, without
requiring any additional training. In order to extend this capability to a
multi-modal setting (i.e. speech and language), this paper introduces the Seal
model, an abbreviation for speech language model. It incorporates a novel
alignment method, in which Kullback-Leibler divergence loss is performed to
train a projector that bridges a frozen speech encoder with a frozen language
model decoder. The resulting Seal model exhibits robust performance as a
few-shot learner on two speech understanding tasks. Additionally, consistency
experiments are conducted to validate its robustness on different pre-trained
language models.
