  We introduce a novel LLM based solution design approach that utilizes
combinatorial optimization and sampling. Specifically, a set of factors that
influence the quality of the solution are identified. They typically include
factors that represent prompt types, LLM inputs alternatives, and parameters
governing the generation and design alternatives. Identifying the factors that
govern the LLM solution quality enables the infusion of subject matter expert
knowledge. Next, a set of interactions between the factors are defined and
combinatorial optimization is used to create a small subset $P$ that ensures
all desired interactions occur in $P$. Each element $p \in P$ is then developed
into an appropriate benchmark. Applying the alternative solutions on each
combination, $p \in P$ and evaluating the results facilitate the design of a
high quality LLM solution pipeline. The approach is especially applicable when
the design and evaluation of each benchmark in $P$ is time-consuming and
involves manual steps and human evaluation. Given its efficiency the approach
can also be used as a baseline to compare and validate an autoML approach that
searches over the factors governing the solution.
