  The increasing use of tools and solutions based on Large Language Models
(LLMs) for various tasks in the medical domain has become a prominent trend.
Their use in this highly critical and sensitive domain has thus raised
important questions about their robustness, especially in response to
variations in input, and the reliability of the generated outputs. This study
addresses these questions by constructing a textual dataset based on the
ICD-10-CM code descriptions, widely used in US hospitals and containing many
clinical terms, and their easily reproducible rephrasing. We then benchmarked
existing embedding models, either generalist or specialized in the clinical
domain, in a semantic search task where the goal was to correctly match the
rephrased text to the original description. Our results showed that generalist
models performed better than clinical models, suggesting that existing clinical
specialized models are more sensitive to small changes in input that confuse
them. The highlighted problem of specialized models may be due to the fact that
they have not been trained on sufficient data, and in particular on datasets
that are not diverse enough to have a reliable global language understanding,
which is still necessary for accurate handling of medical documents.
