  Large language models (LLMs) have led to a surge in collaborative writing
with model assistance. As different users incorporate suggestions from the same
model, there is a risk of decreased diversity in the produced content,
potentially limiting diverse perspectives in public discourse. In this work, we
measure the impact of co-writing on diversity via a controlled experiment,
where users write argumentative essays in three setups -- using a base LLM
(GPT3), a feedback-tuned LLM (InstructGPT), and writing without model help. We
develop a set of diversity metrics and find that writing with InstructGPT (but
not the GPT3) results in a statistically significant reduction in diversity.
Specifically, it increases the similarity between the writings of different
authors and reduces the overall lexical and content diversity. We additionally
find that this effect is mainly attributable to InstructGPT contributing less
diverse text to co-written essays. In contrast, the user-contributed text
remains unaffected by model collaboration. This suggests that the recent
improvement in generation quality from adapting models to human feedback might
come at the cost of more homogeneous and less diverse content.
