  Answering real-world complex queries, such as complex product search, often
requires accurate retrieval from semi-structured knowledge bases that involve
blend of unstructured (e.g., textual descriptions of products) and structured
(e.g., entity relations of products) information. However, previous works have
mostly studied textual and relational retrieval tasks as separate topics. To
address the gap, we develop STARK, a large-scale Semi-structure retrieval
benchmark on Textual and Relational K nowledge Bases. Our benchmark covers
three domains/datasets: product search, academic paper search, and queries in
precision medicine. We design a novel pipeline to synthesize realistic user
queries that integrate diverse relational information and complex textual
properties, together with their ground-truth answers (items). We conduct
rigorous human evaluation to validate the quality of our synthesized queries.
We further enhance the benchmark with high-quality human-generated queries to
provide an authentic reference. STARK serves as a comprehensive testbed for
evaluating the performance of retrieval systems driven by large language models
(LLMs). Our experiments suggest that STARK presents significant challenges to
the current retrieval and LLM systems, indicating the demand for building more
capable retrieval systems. The benchmark data and code are available on
https://github.com/snap-stanford/stark.
