  Much recent effort has been devoted to creating large-scale language models.
Nowadays, the most prominent approaches are based on deep neural networks, such
as BERT. However, they lack transparency and interpretability, and are often
seen as black boxes. This affects not only their applicability in downstream
tasks but also the comparability of different architectures or even of the same
model trained using different corpora or hyperparameters. In this paper, we
propose a set of intrinsic evaluation tasks that inspect the linguistic
information encoded in models developed for Brazilian Portuguese. These tasks
are designed to evaluate how different language models generalise information
related to grammatical structures and multiword expressions (MWEs), thus
allowing for an assessment of whether the model has learned different
linguistic phenomena. The dataset that was developed for these tasks is
composed of a series of sentences with a single masked word and a cue phrase
that helps in narrowing down the context. This dataset is divided into MWEs and
grammatical structures, and the latter is subdivided into 6 tasks: impersonal
verbs, subject agreement, verb agreement, nominal agreement, passive and
connectors. The subset for MWEs was used to test BERTimbau Large, BERTimbau
Base and mBERT. For the grammatical structures, we used only BERTimbau Large,
because it yielded the best results in the MWE task.
