  In November 27, 2018, Amazon Web Services (AWS) released Amazon Comprehend
Medical (ACM), a deep learning based system that automatically extracts
clinical concepts (which include anatomy, medical conditions, protected health
information (PH)I, test names, treatment names, and medical procedures, and
medications) from clinical text notes. Uptake and trust in any new data product
relies on independent validation across benchmark datasets and tools to
establish and confirm expected quality of results. This work focuses on the
medication extraction task, and particularly, ACM was evaluated using the
official test sets from the 2009 i2b2 Medication Extraction Challenge and 2018
n2c2 Track 2: Adverse Drug Events and Medication Extraction in EHRs. Overall,
ACM achieved F-scores of 0.768 and 0.828. These scores ranked the lowest when
compared to the three best systems in the respective challenges. To further
establish the generalizability of its medication extraction performance, a set
of random internal clinical text notes from NYU Langone Medical Center were
also included in this work. And in this corpus, ACM garnered an F-score of
0.753.
