  This paper introduces a novel approach called sentence-wise speech
summarization (Sen-SSum), which generates text summaries from a spoken document
in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of
automatic speech recognition (ASR) with the conciseness of speech
summarization. To explore this approach, we present two datasets for Sen-SSum:
Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of
Transformer-based models: 1) cascade models that combine ASR and strong text
summarization models, and 2) end-to-end (E2E) models that directly convert
speech into a text summary. While E2E models are appealing to develop
compute-efficient models, they perform worse than cascade models. Therefore, we
propose knowledge distillation for E2E models using pseudo-summaries generated
by the cascade models. Our experiments show that this proposed knowledge
distillation effectively improves the performance of the E2E model on both
datasets.
