  This research introduces STAR, a sociotechnical framework that improves on
current best practices for red teaming safety of large language models. STAR
makes two key contributions: it enhances steerability by generating
parameterised instructions for human red teamers, leading to improved coverage
of the risk surface. Parameterised instructions also provide more detailed
insights into model failures at no increased cost. Second, STAR improves signal
quality by matching demographics to assess harms for specific groups, resulting
in more sensitive annotations. STAR further employs a novel step of arbitration
to leverage diverse viewpoints and improve label reliability, treating
disagreement not as noise but as a valuable contribution to signal quality.
