  In recent years, the surge in unstructured data analysis, facilitated by
advancements in Machine Learning (ML), has prompted diverse approaches for
handling images, text documents, and videos. Analysts, leveraging ML models,
can extract meaningful information from unstructured data and store it in
relational databases, allowing the execution of SQL queries for further
analysis. Simultaneously, vector databases have emerged, embedding unstructured
data for efficient top-k queries based on textual queries. This paper
introduces a novel framework SSQL - Semantic SQL that utilizes these two
approaches, enabling the incorporation of semantic queries within SQL
statements. Our approach extends SQL queries with dedicated keywords for
specifying semantic queries alongside predicates related to ML model results
and metadata. Our experimental results show that using just semantic queries
fails catastrophically to answer count and spatial queries in more than 60% of
the cases. Our proposed method jointly optimizes the queries containing both
semantic predicates and predicates on structured tables, such as those
generated by ML models or other metadata. Further, to improve the query
results, we incorporated human-in-the-loop feedback to determine the optimal
similarity score threshold for returning results.
