  The rapid advancement of artificial intelligence (AI) systems suggests that
artificial general intelligence (AGI) systems may soon arrive. Many researchers
are concerned that AIs and AGIs will harm humans via intentional misuse
(AI-misuse) or through accidents (AI-accidents). In respect of AI-accidents,
there is an increasing effort focused on developing algorithms and paradigms
that ensure AI systems are aligned to what humans intend, e.g. AI systems that
yield actions or recommendations that humans might judge as consistent with
their intentions and goals. Here we argue that alignment to human intent is
insufficient for safe AI systems and that preservation of long-term agency of
humans may be a more robust standard, and one that needs to be separated
explicitly and a priori during optimization. We argue that AI systems can
reshape human intention and discuss the lack of biological and psychological
mechanisms that protect humans from loss of agency. We provide the first formal
definition of agency-preserving AI-human interactions which focuses on
forward-looking agency evaluations and argue that AI systems - not humans -
must be increasingly tasked with making these evaluations. We show how agency
loss can occur in simple environments containing embedded agents that use
temporal-difference learning to make action recommendations. Finally, we
propose a new area of research called "agency foundations" and pose four
initial topics designed to improve our understanding of agency in AI-human
interactions: benevolent game theory, algorithmic foundations of human rights,
mechanistic interpretability of agency representation in neural-networks and
reinforcement learning from internal states.
