  With the wide adoption of black-box models, instance-based \emph{post hoc}
explanation tools, such as LIME and SHAP became increasingly popular. These
tools produce explanations, pinpointing contributions of key features
associated with a given prediction. However, the obtained explanations remain
at the raw feature level and are not necessarily understandable by a human
expert without extensive domain knowledge. We propose ReEx (Reasoning with
Explanations), a method applicable to explanations generated by arbitrary
instance-level explainers, such as SHAP. By using background knowledge in the
form of ontologies, ReEx generalizes instance explanations in a least general
generalization-like manner. The resulting symbolic descriptions are specific
for individual classes and offer generalizations based on the explainer's
output. The derived semantic explanations are potentially more informative, as
they describe the key attributes in the context of more general background
knowledge, e.g., at the biological process level. We showcase ReEx's
performance on nine biological data sets, showing that compact, semantic
explanations can be obtained and are more informative than generic ontology
mappings that link terms directly to feature names. ReEx is offered as a
simple-to-use Python library and is compatible with tools such as SHAP and
similar. To our knowledge, this is one of the first methods that directly
couples semantic reasoning with contemporary model explanation methods. This
paper is a preprint. Full version's doi is: 10.1109/SAMI50585.2021.9378668
