  We present CrystalBox, a novel, model-agnostic, posthoc explainability
framework for Deep Reinforcement Learning (DRL) controllers in the large family
of input-driven environments which includes computer systems. We combine the
natural decomposability of reward functions in input-driven environments with
the explanatory power of decomposed returns. We propose an efficient algorithm
to generate future-based explanations across both discrete and continuous
control environments. Using applications such as adaptive bitrate streaming and
congestion control, we demonstrate CrystalBox's capability to generate
high-fidelity explanations. We further illustrate its higher utility across
three practical use cases: contrastive explanations, network observability, and
guided reward design, as opposed to prior explainability techniques that
identify salient features.
