  We introduce the Consistent Reasoning Paradox (CRP). Consistent reasoning,
which lies at the core of human intelligence, is the ability to handle tasks
that are equivalent, yet described by different sentences ('Tell me the time!'
and 'What is the time?'). The CRP asserts that consistent reasoning implies
fallibility -- in particular, human-like intelligence in AI necessarily comes
with human-like fallibility. Specifically, it states that there are problems,
e.g. in basic arithmetic, where any AI that always answers and strives to mimic
human intelligence by reasoning consistently will hallucinate (produce wrong,
yet plausible answers) infinitely often. The paradox is that there exists a
non-consistently reasoning AI (which therefore cannot be on the level of human
intelligence) that will be correct on the same set of problems. The CRP also
shows that detecting these hallucinations, even in a probabilistic sense, is
strictly harder than solving the original problems, and that there are problems
that an AI may answer correctly, but it cannot provide a correct logical
explanation for how it arrived at the answer. Therefore, the CRP implies that
any trustworthy AI (i.e., an AI that never answers incorrectly) that also
reasons consistently must be able to say 'I don't know'. Moreover, this can
only be done by implicitly computing a new concept that we introduce, termed
the 'I don't know' function -- something currently lacking in modern AI. In
view of these insights, the CRP also provides a glimpse into the behaviour of
Artificial General Intelligence (AGI). An AGI cannot be 'almost sure', nor can
it always explain itself, and therefore to be trustworthy it must be able to
say 'I don't know'.
