  The rapid emergence of generative Language Models (LMs) has led to growing
concern about the impacts that their unexamined adoption may have on the social
well-being of diverse user groups. Meanwhile, LMs are increasingly being
adopted in K-20 schools and one-on-one student settings with minimal
investigation of potential harms associated with their deployment. Motivated in
part by real-world/everyday use cases (e.g., an AI writing assistant) this
paper explores the potential psychosocial harms of stories generated by five
leading LMs in response to open-ended prompting. We extend findings of
stereotyping harms analyzing a total of 150K 100-word stories related to
student classroom interactions. Examining patterns in LM-generated character
demographics and representational harms (i.e., erasure, subordination, and
stereotyping) we highlight particularly egregious vignettes, illustrating the
ways LM-generated outputs may influence the experiences of users with
marginalized and minoritized identities, and emphasizing the need for a
critical understanding of the psychosocial impacts of generative AI tools when
deployed and utilized in diverse social contexts.
