  There is overwhelming evidence that human intelligence is a product of
Darwinian evolution. Investigating the consequences of self-modification, and
more precisely, the consequences of utility function self-modification, leads
to the stronger claim that not only human, but any form of intelligence is
ultimately only possible within evolutionary processes. Human-designed
artificial intelligences can only remain stable until they discover how to
manipulate their own utility function. By definition, a human designer cannot
prevent a superhuman intelligence from modifying itself, even if protection
mechanisms against this action are put in place. Without evolutionary pressure,
sufficiently advanced artificial intelligences become inert by simplifying
their own utility function. Within evolutionary processes, the implicit utility
function is always reducible to persistence, and the control of superhuman
intelligences embedded in evolutionary processes is not possible. Mechanisms
against utility function self-modification are ultimately futile. Instead,
scientific effort toward the mitigation of existential risks from the
development of superintelligences should be in two directions: understanding
consciousness, and the complex dynamics of evolutionary systems.
