  A common approach to explaining NLP models is to use importance measures that
express which tokens are important for a prediction. Unfortunately, such
explanations are often wrong despite being persuasive. Therefore, it is
essential to measure their faithfulness. One such metric is if tokens are truly
important, then masking them should result in worse model performance. However,
token masking introduces out-of-distribution issues, and existing solutions
that address this are computationally expensive and employ proxy models.
Furthermore, other metrics are very limited in scope. This work proposes an
inherently faithfulness measurable model that addresses these challenges. This
is achieved using a novel fine-tuning method that incorporates masking, such
that masking tokens become in-distribution by design. This differs from
existing approaches, which are completely model-agnostic but are inapplicable
in practice. We demonstrate the generality of our approach by applying it to 16
different datasets and validate it using statistical in-distribution tests. The
faithfulness is then measured with 9 different importance measures. Because
masking is in-distribution, importance measures that themselves use masking
become consistently more faithful. Additionally, because the model makes
faithfulness cheap to measure, we can optimize explanations towards maximal
faithfulness; thus, our model becomes indirectly inherently explainable.
