  Collaborative problem solving (CPS) in teams is tightly coupled with the
creation of shared meaning between participants in a situated, collaborative
task. In this work, we assess the quality of different utterance segmentation
techniques as an aid in annotating CPS. We (1) manually transcribe utterances
in a dataset of triads collaboratively solving a problem involving dialogue and
physical object manipulation, (2) annotate collaborative moves according to
these gold-standard transcripts, and then (3) apply these annotations to
utterances that have been automatically segmented using toolkits from Google
and OpenAI's Whisper. We show that the oracle utterances have minimal
correspondence to automatically segmented speech, and that automatically
segmented speech using different segmentation methods is also inconsistent. We
also show that annotating automatically segmented speech has distinct
implications compared with annotating oracle utterances--since most annotation
schemes are designed for oracle cases, when annotating automatically-segmented
utterances, annotators must invoke other information to make arbitrary
judgments which other annotators may not replicate. We conclude with a
discussion of how future annotation specs can account for these needs.
