  Background: Transformer-based language models have shown strong performance
on many Natural LanguageProcessing (NLP) tasks. Masked Language Models (MLMs)
attract sustained interest because they can be adaptedto different languages
and sub-domains through training or fine-tuning on specific corpora while
remaining lighterthan modern Large Language Models (LLMs). Recently, several
MLMs have been released for the biomedicaldomain in French, and experiments
suggest that they outperform standard French counterparts. However,
nosystematic evaluation comparing all models on the same corpora is available.
Objective: This paper presentsan evaluation of masked language models for
biomedical French on the task of clinical named entity recognition.Material and
methods: We evaluate biomedical models CamemBERT-bio and DrBERT and compare
them tostandard French models CamemBERT, FlauBERT and FrALBERT as well as
multilingual mBERT using three publicallyavailable corpora for clinical named
entity recognition in French. The evaluation set-up relies on
gold-standardcorpora as released by the corpus developers. Results: Results
suggest that CamemBERT-bio outperformsDrBERT consistently while FlauBERT offers
competitive performance and FrAlBERT achieves the lowest carbonfootprint.
Conclusion: This is the first benchmark evaluation of biomedical masked
language models for Frenchclinical entity recognition that compares model
performance consistently on nested entity recognition using metricscovering
performance and environmental impact.
