  Machine learning is becoming increasingly important to control the behavior
of safety and financially critical components in sophisticated environments,
where the inability to understand learned components in general, and neural
nets in particular, poses serious obstacles to their adoption. Explainability
and interpretability methods for learned systems have gained considerable
academic attention, but the focus of current approaches on only one aspect of
explanation, at a fixed level of abstraction, and limited if any formal
guarantees, prevents those explanations from being digestible by the relevant
stakeholders (e.g., end users, certification authorities, engineers) with their
diverse backgrounds and situation-specific needs. We introduce Fanoos, a
framework for combining formal verification techniques, heuristic search, and
user interaction to explore explanations at the desired level of granularity
and fidelity. We demonstrate the ability of Fanoos to produce and adjust the
abstractness of explanations in response to user requests on a learned
controller for an inverted double pendulum and on a learned CPU usage model.
