  Relation extraction in the biomedical domain is challenging due to the lack
of labeled data and high annotation costs, needing domain experts. Distant
supervision is commonly used to tackle the scarcity of annotated data by
automatically pairing knowledge graph relationships with raw texts. Such a
pipeline is prone to noise and has added challenges to scale for covering a
large number of biomedical concepts. We investigated existing broad-coverage
distantly supervised biomedical relation extraction benchmarks and found a
significant overlap between training and test relationships ranging from 26% to
86%. Furthermore, we noticed several inconsistencies in the data construction
process of these benchmarks, and where there is no train-test leakage, the
focus is on interactions between narrower entity types. This work presents a
more accurate benchmark MedDistant19 for broad-coverage distantly supervised
biomedical relation extraction that addresses these shortcomings and is
obtained by aligning the MEDLINE abstracts with the widely used SNOMED Clinical
Terms knowledge base. Lacking thorough evaluation with domain-specific language
models, we also conduct experiments validating general domain relation
extraction findings to biomedical relation extraction.
