  Decision support systems based on clinical notes have the potential to
improve patient care by pointing doctors towards overseen risks. Predicting a
patient's outcome is an essential part of such systems, for which the use of
deep neural networks has shown promising results. However, the patterns learned
by these networks are mostly opaque and previous work revealed flaws regarding
the reproduction of unintended biases. We thus introduce an extendable testing
framework that evaluates the behavior of clinical outcome models regarding
changes of the input. The framework helps to understand learned patterns and
their influence on model decisions. In this work, we apply it to analyse the
change in behavior with regard to the patient characteristics gender, age and
ethnicity. Our evaluation of three current clinical NLP models demonstrates the
concrete effects of these characteristics on the models' decisions. They show
that model behavior varies drastically even when fine-tuned on the same data
and that allegedly best-performing models have not always learned the most
medically plausible patterns.
