  Recent advances in the development of pre-trained Spanish language models has
led to significant progress in many Natural Language Processing (NLP) tasks,
such as question answering. However, the lack of efficient models imposes a
barrier for the adoption of such models in resource-constrained environments.
Therefore, smaller distilled models for the Spanish language could be proven to
be highly scalable and facilitate their further adoption on a variety of tasks
and scenarios. In this work, we take one step in this direction by developing
SpanishTinyRoBERTa, a compressed language model based on RoBERTa for efficient
question answering in Spanish. To achieve this, we employ knowledge
distillation from a large model onto a lighter model that allows for a wider
implementation, even in areas with limited computational resources, whilst
attaining negligible performance sacrifice. Our experiments show that the dense
distilled model can still preserve the performance of its larger counterpart,
while significantly increasing inference speedup. This work serves as a
starting point for further research and investigation of model compression
efforts for Spanish language models across various NLP tasks.
