  Language models learn and represent language differently than humans; they
learn the form and not the meaning. Thus, to assess the success of language
model explainability, we need to consider the impact of its divergence from a
user's mental model of language. In this position paper, we argue that in order
to avoid harmful rationalization and achieve truthful understanding of language
models, explanation processes must satisfy three main conditions: (1)
explanations have to truthfully represent the model behavior, i.e., have a high
fidelity; (2) explanations must be complete, as missing information distorts
the truth; and (3) explanations have to take the user's mental model into
account, progressively verifying a person's knowledge and adapting their
understanding. We introduce a decision tree model to showcase potential reasons
why current explanations fail to reach their objectives. We further emphasize
the need for human-centered design to explain the model from multiple
perspectives, progressively adapting explanations to changing user
expectations.
