  In this paper, we present our progress in pretraining Czech monolingual audio
transformers from a large dataset containing more than 80 thousand hours of
unlabeled speech, and subsequently fine-tuning the model on automatic speech
recognition tasks using a combination of in-domain data and almost 6 thousand
hours of out-of-domain transcribed speech. We are presenting a large palette of
experiments with various fine-tuning setups evaluated on two public datasets
(CommonVoice and VoxPopuli) and one extremely challenging dataset from the
MALACH project. Our results show that monolingual Wav2Vec 2.0 models are robust
ASR systems, which can take advantage of large labeled and unlabeled datasets
and successfully compete with state-of-the-art LVCSR systems. Moreover, Wav2Vec
models proved to be good zero-shot learners when no training data are available
for the target ASR task.
