  Deep Learning of neural networks has progressively become more prominent in
healthcare with models reaching, or even surpassing, expert accuracy levels.
However, these success stories are tainted by concerning reports on the lack of
model transparency and bias against some medical conditions or patients'
sub-groups. Explainable methods are considered the gateway to alleviate many of
these concerns. In this study we demonstrate that the generated explanations
are volatile to changes in model training that are perpendicular to the
classification task and model structure. This raises further questions about
trust in deep learning models for healthcare. Mainly, whether the models
capture underlying causal links in the data or just rely on spurious
correlations that are made visible via explanation methods. We demonstrate that
the output of explainability methods on deep neural networks can vary
significantly by changes of hyper-parameters, such as the random seed or how
the training set is shuffled. We introduce a measure of explanation consistency
which we use to highlight the identified problems on the MIMIC-CXR dataset. We
find explanations of identical models but with different training setups have a
low consistency: $\approx$ 33% on average. On the contrary, kernel methods are
robust against any orthogonal changes, with explanation consistency at 94%. We
conclude that current trends in model explanation are not sufficient to
mitigate the risks of deploying models in real life healthcare applications.
