  We introduce Doctor XAvIer, a BERT-based diagnostic system that extracts
relevant clinical data from transcribed patient-doctor dialogues and explains
predictions using feature attribution methods. We present a novel performance
plot and evaluation metric for feature attribution methods: Feature Attribution
Dropping (FAD) curve and its Normalized Area Under the Curve (N-AUC). FAD curve
analysis shows that integrated gradients outperforms Shapley values in
explaining diagnosis classification. Doctor XAvIer outperforms the baseline
with 0.97 F1-score in named entity recognition and symptom pertinence
classification and 0.91 F1-score in diagnosis classification.
