  Text-to-SQL, the task of translating natural language questions into SQL
queries, is part of various business processes. Its automation, which is an
emerging challenge, will empower software practitioners to seamlessly interact
with relational databases using natural language, thereby bridging the gap
between business needs and software capabilities. In this paper, we consider
Large Language Models (LLMs), which have achieved state of the art for various
NLP tasks. Specifically, we benchmark Text-to-SQL performance, the evaluation
methodologies, as well as input optimization (e.g., prompting). In light of the
empirical observations that we have made, we propose two novel metrics that
were designed to adequately measure the similarity between SQL queries.
Overall, we share with the community various findings, notably on how to select
the right LLM on Text-to-SQL tasks. We further demonstrate that a tree-based
edit distance constitutes a reliable metric for assessing the similarity
between generated SQL queries and the oracle for benchmarking Text2SQL
approaches. This metric is important as it relieves researchers from the need
to perform computationally expensive experiments such as executing generated
queries as done in prior works. Our work implements financial domain use cases
and, therefore contributes to the advancement of Text2SQL systems and their
practical adoption in this domain.
