  With the methodological support of probing (or diagnostic classification),
recent studies have demonstrated that Transformers encode syntactic and
semantic information to some extent. Following this line of research, this
paper aims at taking semantic probing to an abstraction extreme with the goal
of answering the following research question: can contemporary
Transformer-based models reflect an underlying Foundational Ontology? To this
end, we present a systematic Foundational Ontology (FO) probing methodology to
investigate whether Transformers-based models encode abstract semantic
information. Following different pre-training and fine-tuning regimes, we
present an extensive evaluation of a diverse set of large-scale language models
over three distinct and complementary FO tagging experiments. Specifically, we
present and discuss the following conclusions: (1) The probing results indicate
that Transformer-based models incidentally encode information related to
Foundational Ontologies during the pre-training pro-cess; (2) Robust FO taggers
(accuracy of 90 percent)can be efficiently built leveraging on this knowledge.
