  With the aim to provide teachers with more specific, frequent, and actionable
feedback about their teaching, we explore how Large Language Models (LLMs) can
be used to estimate ``Instructional Support'' domain scores of the CLassroom
Assessment Scoring System (CLASS), a widely used observation protocol. We
design a machine learning architecture that uses either zero-shot prompting of
Meta's Llama2, and/or a classic Bag of Words (BoW) model, to classify
individual utterances of teachers' speech (transcribed automatically using
OpenAI's Whisper) for the presence of Instructional Support. Then, these
utterance-level judgments are aggregated over a 15-min observation session to
estimate a global CLASS score. Experiments on two CLASS-coded datasets of
toddler and pre-kindergarten classrooms indicate that (1) automatic CLASS
Instructional Support estimation accuracy using the proposed method (Pearson
$R$ up to $0.48$) approaches human inter-rater reliability (up to $R=0.55$);
(2) LLMs generally yield slightly greater accuracy than BoW for this task,
though the best models often combined features extracted from both LLM and BoW;
and (3) for classifying individual utterances, there is still room for
improvement of automated methods compared to human-level judgments. Finally,
(4) we illustrate how the model's outputs can be visualized at the utterance
level to provide teachers with explainable feedback on which utterances were
most positively or negatively correlated with specific CLASS dimensions.
