  Much work aims to explain a model's prediction on a static input. We consider
explanations in a temporal setting where a stateful dynamical model produces a
sequence of risk estimates given an input at each time step. When the estimated
risk increases, the goal of the explanation is to attribute the increase to a
few relevant inputs from the past. While our formal setup and techniques are
general, we carry out an in-depth case study in a clinical setting. The goal
here is to alert a clinician when a patient's risk of deterioration rises. The
clinician then has to decide whether to intervene and adjust the treatment.
Given a potentially long sequence of new events since she last saw the patient,
a concise explanation helps her to quickly triage the alert. We develop methods
to lift static attribution techniques to the dynamical setting, where we
identify and address challenges specific to dynamics. We then experimentally
assess the utility of different explanations of clinical alerts through expert
evaluation.
