  Large Language Models (LLMs) are currently under exploration for various
tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT),
and even End-to-End Speech Translation (ST). In this paper, we present KIT's
offline submission in the constrained + LLM track by incorporating recently
proposed techniques that can be added to any cascaded speech translation.
Specifically, we integrate
Mistral-7B\footnote{mistralai/Mistral-7B-Instruct-v0.1} into our system to
enhance it in two ways. Firstly, we refine the ASR outputs by utilizing the
N-best lists generated by our system and fine-tuning the LLM to predict the
transcript accurately. Secondly, we refine the MT outputs at the document level
by fine-tuning the LLM, leveraging both ASR and MT predictions to improve
translation quality. We find that integrating the LLM into the ASR and MT
systems results in an absolute improvement of $0.3\%$ in Word Error Rate and
$0.65\%$ in COMET for tst2019 test set. In challenging test sets with
overlapping speakers and background noise, we find that integrating LLM is not
beneficial due to poor ASR performance. Here, we use ASR with chunked long-form
decoding to improve context usage that may be unavailable when transcribing
with Voice Activity Detection segmentation alone.
