  Recent advances in LLMs have sparked a debate on whether they understand
text. In this position paper, we argue that opponents in this debate hold
different definitions for understanding, and particularly differ in their view
on the role of consciousness. To substantiate this claim, we propose a thought
experiment involving an open-source chatbot $Z$ which excels on every possible
benchmark, seemingly without subjective experience. We ask whether $Z$ is
capable of understanding, and show that different schools of thought within
seminal AI research seem to answer this question differently, uncovering their
terminological disagreement. Moving forward, we propose two distinct working
definitions for understanding which explicitly acknowledge the question of
consciousness, and draw connections with a rich literature in philosophy,
psychology and neuroscience.
