  Recent studies have made some progress in refining end-to-end (E2E) speech
recognition encoders by applying Connectionist Temporal Classification (CTC)
loss to enhance named entity recognition within transcriptions. However, these
methods have been constrained by their exclusive use of the ASCII character
set, allowing only a limited array of semantic labels. We propose 1SPU, a
1-step Speech Processing Unit which can recognize speech events (e.g: speaker
change) or an NL event (Intent, Emotion) while also transcribing vocal content.
It extends the E2E automatic speech recognition (ASR) system's vocabulary by
adding a set of unused placeholder symbols, conceptually akin to the <pad>
tokens used in sequence modeling. These placeholders are then assigned to
represent semantic events (in form of tags) and are integrated into the
transcription process as distinct tokens.
  We demonstrate notable improvements on the SLUE benchmark and yields results
that are on par with those for the SLURP dataset. Additionally, we provide a
visual analysis of the system's proficiency in accurately pinpointing
meaningful tokens over time, illustrating the enhancement in transcription
quality through the utilization of supplementary semantic tags.
