Evaluation of a Bi-Directional Methodology for Automated Assessment of
  Compliance to Continuous Application of Clinical Guidelines, in the Type 2
  Diabetes-Management Domain
  We evaluated the DiscovErr system, in which we had previously implemented a
new methodology for assessment of compliance to continuous application of
evidence-based clinical guidelines, based on a bidirectional search from the
guideline objectives to the patient's longitudinal data, and vice versa. We
compared the system comments on 1584 transactions regarding the management,
over a mean of 5.23 years, of 10 randomly selected Type 2 diabetes patients, to
those of two diabetes experts and a senior family practitioner. After providing
their own comments, the experts assessed both the correctness (precision) and
the importance of each of the DiscovErr system comments. The completeness
(recall or coverage) of the system was computed by comparing its comments to
those made by the experts. The system made 279 comments. The experts made 181
unique comments. The completeness of the system was 91% compared to comments
made by at least two experts, and 98% when compared to comments made by all
three. 172 comments were evaluated by the experts for correctness and
importance: All 114 medication-related comments, and a random 35% of the 165
monitoring-related comments. The system's correctness was 81% compared to
comments judged as correct by both diabetes experts, and 91% compared to
comments judged as correct by a diabetes expert and at least as partially
correct by the other. 89% of the comments were judged as important by both
diabetes experts, 8% were judged as important by one expert, 3% were judged as
less important by both experts. The completeness scores of the three experts
(compared to the comments of all experts plus the validated system comments)
were 75%, 60%, and 55%; the experts' correctness scores (compared to their
majority) were respectively 99%, 91%, and 88%. Conclusion: Systems such as
DiscovErr can assess the quality of continuous guideline-based care.
