  At the staggering pace with which the capabilities of large language models
(LLMs) are increasing, creating future-proof evaluation sets to assess their
understanding becomes more and more challenging. In this paper, we propose a
novel paradigm for evaluating LLMs which leverages the idea that correct world
understanding should be consistent across different (Fregean) senses of the
same meaning. Accordingly, we measure understanding not in terms of correctness
but by evaluating consistency across multiple senses that are generated by the
model itself. We showcase our approach by instantiating a test where the
different senses are different languages, hence using multilingual
self-consistency as a litmus test for the model's understanding and
simultaneously addressing the important topic of multilinguality. Taking one of
the latest versions of ChatGPT as our object of study, we evaluate multilingual
consistency for two different tasks across three different languages. We show
that its multilingual consistency is still lacking, and that its task and world
understanding are thus not language-independent. As our approach does not
require any static evaluation corpora in languages other than English, it can
easily and cheaply be extended to different languages and tasks and could
become an integral part of future benchmarking efforts.
