  The neural architectures of language models are becoming increasingly
complex, especially that of Transformers, based on the attention mechanism.
Although their application to numerous natural language processing tasks has
proven to be very fruitful, they continue to be models with little or no
interpretability and explainability. One of the tasks for which they are best
suited is the encoding of the contextual sense of words using contextualized
embeddings. In this paper we propose a transparent, interpretable, and
linguistically motivated strategy for encoding the contextual sense of words by
modeling semantic compositionality. Particular attention is given to dependency
relations and semantic notions such as selection preferences and paradigmatic
classes. A partial implementation of the proposed model is carried out and
compared with Transformer-based architectures for a given semantic task, namely
the similarity calculation of word senses in context. The results obtained show
that it is possible to be competitive with linguistically motivated models
instead of using the black boxes underlying complex neural architectures.
