  As AI systems become an increasing part of people's everyday lives, it
becomes ever more important that they understand people's ethical norms.
Motivated by descriptive ethics, a field of study that focuses on people's
descriptive judgments rather than theoretical prescriptions on morality, we
investigate a novel, data-driven approach to machine ethics.
  We introduce Scruples, the first large-scale dataset with 625,000 ethical
judgments over 32,000 real-life anecdotes. Each anecdote recounts a complex
ethical situation, often posing moral dilemmas, paired with a distribution of
judgments contributed by the community members. Our dataset presents a major
challenge to state-of-the-art neural language models, leaving significant room
for improvement. However, when presented with simplified moral situations, the
results are considerably more promising, suggesting that neural models can
effectively learn simpler ethical building blocks.
  A key take-away of our empirical analysis is that norms are not always
clean-cut; many situations are naturally divisive. We present a new method to
estimate the best possible performance on such tasks with inherently diverse
label distributions, and explore likelihood functions that separate intrinsic
from model uncertainty.
