  Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise
problematic behavior, such as helping to commit crimes or producing racist
text. One approach to fine-tuning, called reinforcement learning from human
feedback, learns from humans' expressed preferences over multiple outputs.
Another approach is constitutional AI, in which the input from humans is a list
of high-level principles. But how do we deal with potentially diverging input
from humans? How can we aggregate the input into consistent data about
"collective" preferences or otherwise use it to make collective choices about
model behavior? In this paper, we argue that the field of social choice is well
positioned to address these questions, and we discuss ways forward for this
agenda, drawing on discussions in a recent workshop on Social Choice for AI
Ethics and Safety held in Berkeley, CA, USA in December 2023.
