  The overarching goal of Explainable AI is to develop systems that not only
exhibit intelligent behaviours, but also are able to explain their rationale
and reveal insights. In explainable machine learning, methods that produce a
high level of prediction accuracy as well as transparent explanations are
valuable. In this work, we present an explainable classification method. Our
method works by first constructing a symbolic Knowledge Base from the training
data, and then performing probabilistic inferences on such Knowledge Base with
linear programming. Our approach achieves a level of learning performance
comparable to that of traditional classifiers such as random forests, support
vector machines and neural networks. It identifies decisive features that are
responsible for a classification as explanations and produces results similar
to the ones found by SHAP, a state of the art Shapley Value based method. Our
algorithms perform well on a range of synthetic and non-synthetic data sets.
