  Diabetic eye disease is a major cause of blindness worldwide. The ability to
monitor relevant clinical trajectories and detect lapses in care is critical to
managing the disease and preventing blindness. Alas, much of the information
necessary to support these goals is found only in the free text of the
electronic medical record. To fill this information gap, we introduce a system
for extracting evidence from clinical text of 19 clinical concepts related to
diabetic eye disease and inferring relevant attributes for each. In developing
this ophthalmology phenotyping system, we are also afforded a unique
opportunity to evaluate the effectiveness of clinical language models at
adapting to new clinical domains. Across multiple training paradigms, we find
that BERT language models pretrained on out-of-distribution clinical data offer
no significant improvement over BERT language models pretrained on non-clinical
data for our domain. Our study tempers recent claims that language models
pretrained on clinical data are necessary for clinical NLP tasks and highlights
the importance of not treating clinical language data as a single homogeneous
domain.
