  We aim at improving spoken language modeling (LM) using very large amount of
automatically transcribed speech. We leverage the INA (French National
Audiovisual Institute) collection and obtain 19GB of text after applying ASR on
350,000 hours of diverse TV shows. From this, spoken language models are
trained either by fine-tuning an existing LM (FlauBERT) or through training a
LM from scratch. New models (FlauBERT-Oral) are shared with the community and
evaluated for 3 downstream tasks: spoken language understanding, classification
of TV shows and speech syntactic parsing. Results show that FlauBERT-Oral can
be beneficial compared to its initial FlauBERT version demonstrating that,
despite its inherent noisy nature, ASR-generated text can be used to build
spoken language models.
