  Despite the advances in digital healthcare systems offering curated
structured knowledge, much of the critical information still lies in large
volumes of unlabeled and unstructured clinical texts. These texts, which often
contain protected health information (PHI), are exposed to information
extraction tools for downstream applications, risking patient identification.
Existing works in de-identification rely on using large-scale annotated corpora
in English, which often are not suitable in real-world multilingual settings.
Pre-trained language models (LM) have shown great potential for cross-lingual
transfer in low-resource settings. In this work, we empirically show the
few-shot cross-lingual transfer property of LMs for named entity recognition
(NER) and apply it to solve a low-resource and real-world challenge of
code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke
domain. We annotate a gold evaluation dataset to assess few-shot setting
performance where we only use a few hundred labeled examples for training. Our
model improves the zero-shot F1-score from 73.7% to 91.2% on the gold
evaluation set when adapting Multilingual BERT (mBERT) (Devlin et al., 2019)
from the MEDDOCAN (Marimon et al., 2019) corpus with our few-shot cross-lingual
target corpus. When generalized to an out-of-sample test set, the best model
achieves a human-evaluation F1-score of 97.2%.
