  With the growing prevalence of large language models, it is increasingly
common to annotate datasets for machine learning using pools of crowd raters.
However, these raters often work in isolation as individual crowdworkers. In
this work, we regard annotation not merely as inexpensive, scalable labor, but
rather as a nuanced interpretative effort to discern the meaning of what is
being said in a text. We describe a novel, collaborative, and iterative
annotator-in-the-loop methodology for annotation, resulting in a 'Bridging
Benchmark Dataset' of comments relevant to bridging divides, annotated from
11,973 textual posts in the Civil Comments dataset. The methodology differs
from popular anonymous crowd-rating annotation processes due to its use of an
in-depth, iterative engagement with seven US-based raters to (1)
collaboratively refine the definitions of the to-be-annotated concepts and then
(2) iteratively annotate complex social concepts, with check-in meetings and
discussions. This approach addresses some shortcomings of current anonymous
crowd-based annotation work, and we present empirical evidence of the
performance of our annotation process in the form of inter-rater reliability.
Our findings indicate that collaborative engagement with annotators can enhance
annotation methods, as opposed to relying solely on isolated work conducted
remotely. We provide an overview of the input texts, attributes, and annotation
process, along with the empirical results and the resulting benchmark dataset,
categorized according to the following attributes: Alienation, Compassion,
Reasoning, Curiosity, Moral Outrage, and Respect.
