  As AI systems become increasingly powerful and pervasive, there are growing
concerns about machines' morality or a lack thereof. Yet, teaching morality to
machines is a formidable task, as morality remains among the most intensely
debated questions in humanity, let alone for AI. Existing AI systems deployed
to millions of users, however, are already making decisions loaded with moral
implications, which poses a seemingly impossible challenge: teaching machines
moral sense, while humanity continues to grapple with it.
  To explore this challenge, we introduce Delphi, an experimental framework
based on deep neural networks trained directly to reason about descriptive
ethical judgments, e.g., "helping a friend" is generally good, while "helping a
friend spread fake news" is not. Empirical results shed novel insights on the
promises and limits of machine ethics; Delphi demonstrates strong
generalization capabilities in the face of novel ethical situations, while
off-the-shelf neural network models exhibit markedly poor judgment including
unjust biases, confirming the need for explicitly teaching machines moral
sense.
  Yet, Delphi is not perfect, exhibiting susceptibility to pervasive biases and
inconsistencies. Despite that, we demonstrate positive use cases of imperfect
Delphi, including using it as a component model within other imperfect AI
systems. Importantly, we interpret the operationalization of Delphi in light of
prominent ethical theories, which leads us to important future research
questions.
