  Tsetlin Machines (TMs) capture patterns using conjunctive clauses in
propositional logic, thus facilitating interpretation. However, recent TM-based
approaches mainly rely on inspecting the full range of clauses individually.
Such inspection does not necessarily scale to complex prediction problems that
require a large number of clauses. In this paper, we propose closed-form
expressions for understanding why a TM model makes a specific prediction (local
interpretability). Additionally, the expressions capture the most important
features of the model overall (global interpretability). We further introduce
expressions for measuring the importance of feature value ranges for continuous
features. The expressions are formulated directly from the conjunctive clauses
of the TM, making it possible to capture the role of features in real-time,
also during the learning process as the model evolves. Additionally, from the
closed-form expressions, we derive a novel data clustering algorithm for
visualizing high-dimensional data in three dimensions. Finally, we compare our
proposed approach against SHAP and state-of-the-art interpretable machine
learning techniques. For both classification and regression, our evaluation
show correspondence with SHAP as well as competitive prediction accuracy in
comparison with XGBoost, Explainable Boosting Machines, and Neural Additive
Models.
