  Electronic healthcare records are vital for patient safety as they document
conditions, plans, and procedures in both free text and medical codes. Language
models have significantly enhanced the processing of such records, streamlining
workflows and reducing manual data entry, thereby saving healthcare providers
significant resources. However, the black-box nature of these models often
leaves healthcare professionals hesitant to trust them. State-of-the-art
explainability methods increase model transparency but rely on human-annotated
evidence spans, which are costly. In this study, we propose an approach to
produce plausible and faithful explanations without needing such annotations.
We demonstrate on the automated medical coding task that adversarial robustness
training improves explanation plausibility and introduce AttInGrad, a new
explanation method superior to previous ones. By combining both contributions
in a fully unsupervised setup, we produce explanations of comparable quality,
or better, to that of a supervised approach. We release our code and model
weights.
