  Contextualised word embeddings is a powerful tool to detect contextual
synonyms. However, most of the current state-of-the-art (SOTA) deep learning
concept extraction methods remain supervised and underexploit the potential of
the context. In this paper, we propose a self-supervised pre-training approach
which is able to detect contextual synonyms of concepts being training on the
data created by shallow matching. We apply our methodology in the sparse
multi-class setting (over 15,000 concepts) to extract phenotype information
from electronic health records. We further investigate data augmentation
techniques to address the problem of the class sparsity. Our approach achieves
a new SOTA for the unsupervised phenotype concept annotation on clinical text
on F1 and Recall outperforming the previous SOTA with a gain of up to 4.5 and
4.0 absolute points, respectively. After fine-tuning with as little as 20\% of
the labelled data, we also outperform BioBERT and ClinicalBERT. The extrinsic
evaluation on three ICU benchmarks also shows the benefit of using the
phenotypes annotated by our model as features.
