  Electronic medical records (EMRs) are stored in relational databases. It can
be challenging to access the required information if the user is unfamiliar
with the database schema or general database fundamentals. Hence, researchers
have explored text-to-SQL generation methods that provide healthcare
professionals direct access to EMR data without needing a database expert.
However, currently available datasets have been essentially "solved" with
state-of-the-art models achieving accuracy greater than or near 90%. In this
paper, we show that there is still a long way to go before solving text-to-SQL
generation in the medical domain. To show this, we create new splits of the
existing medical text-to-SQL dataset MIMICSQL that better measure the
generalizability of the resulting models. We evaluate state-of-the-art language
models on our new split showing substantial drops in performance with accuracy
dropping from up to 92% to 28%, thus showing substantial room for improvement.
Moreover, we introduce a novel data augmentation approach to improve the
generalizability of the language models. Overall, this paper is the first step
towards developing more robust text-to-SQL models in the medical
domain.\footnote{The dataset and code will be released upon acceptance.
