  The recent introduction of Transformers language representation models
allowed great improvements in many natural language processing (NLP) tasks.
However, if on one hand the performances achieved by this kind of architectures
are surprising, on the other their usability is limited by the high number of
parameters which constitute their network, resulting in high computational and
memory demands. In this work we present BERTino, a DistilBERT model which
proposes to be the first lightweight alternative to the BERT architecture
specific for the Italian language. We evaluated BERTino on the Italian ISDT,
Italian ParTUT, Italian WikiNER and multiclass classification tasks, obtaining
F1 scores comparable to those obtained by a BERTBASE with a remarkable
improvement in training and inference speed.
