  Conventional conversation assistants extract text transcripts from the speech
signal using automatic speech recognition (ASR) and then predict intent from
the transcriptions. Using end-to-end spoken language understanding (SLU), the
intents of the speaker are predicted directly from the speech signal without
requiring intermediate text transcripts. As a result, the model can optimize
directly for intent classification and avoid cascading errors from ASR. The
end-to-end SLU system also helps in reducing the latency of the intent
prediction model. Although many datasets are available publicly for
text-to-intent tasks, the availability of labeled speech-to-intent datasets is
limited, and there are no datasets available in the Indian accent. In this
paper, we release the Skit-S2I dataset, the first publicly available
Indian-accented SLU dataset in the banking domain in a conversational tonality.
We experiment with multiple baselines, compare different pretrained speech
encoder's representations, and find that SSL pretrained representations perform
slightly better than ASR pretrained representations lacking prosodic features
for speech-to-intent classification. The dataset and baseline code is available
at \url{https://github.com/skit-ai/speech-to-intent-dataset}
