  Large language models (LLMs) can explain their predictions through post-hoc
or Chain-of-Thought (CoT) explanations. But an LLM could make up reasonably
sounding explanations that are unfaithful to its underlying reasoning. Recent
work has designed tests that aim to judge the faithfulness of post-hoc or CoT
explanations. In this work we argue that these faithfulness tests do not
measure faithfulness to the models' inner workings -- but rather their
self-consistency at output level. Our contributions are three-fold: i) We
clarify the status of faithfulness tests in view of model explainability,
characterising them as self-consistency tests instead. This assessment we
underline by ii) constructing a Comparative Consistency Bank for
self-consistency tests that for the first time compares existing tests on a
common suite of 11 open LLMs and 5 tasks -- including iii) our new
self-consistency measure CC-SHAP. CC-SHAP is a fine-grained measure (not a
test) of LLM self-consistency. It compares how a model's input contributes to
the predicted answer and to generating the explanation. Our fine-grained
CC-SHAP metric allows us iii) to compare LLM behaviour when making predictions
and to analyse the effect of other consistency tests at a deeper level, which
takes us one step further towards measuring faithfulness by bringing us closer
to the internals of the model than strictly surface output-oriented tests. Our
code is available at \url{https://github.com/Heidelberg-NLP/CC-SHAP}
