  This work explores speech as a biomarker and investigates the detection of
respiratory insufficiency (RI) by analyzing speech samples. Previous work
\cite{spira2021} constructed a dataset of respiratory insufficiency COVID-19
patient utterances and analyzed it by means of a convolutional neural network
achieving an accuracy of $87.04\%$, validating the hypothesis that one can
detect RI through speech. Here, we study how Transformer neural network
architectures can improve the performance on RI detection. This approach
enables construction of an acoustic model. By choosing the correct pretraining
technique, we generate a self-supervised acoustic model, leading to improved
performance ($96.53\%$) of Transformers for RI detection.
