  The healthcare domain is one of the most exciting application areas for
machine learning, but a lack of model transparency contributes to a lag in
adoption within the industry. In this work, we explore the current art of
explainability and interpretability within a case study in clinical text
classification, using a task of mortality prediction within MIMIC-III clinical
notes. We demonstrate various visualization techniques for fully interpretable
methods as well as model-agnostic post hoc attributions, and we provide a
generalized method for evaluating the quality of explanations using infidelity
and local Lipschitz across model types from logistic regression to BERT
variants. With these metrics, we introduce a framework through which
practitioners and researchers can assess the frontier between a model's
predictive performance and the quality of its available explanations. We make
our code available to encourage continued refinement of these methods.
